Skip to content

Neops Core 2.3.0

Released 2026-10-29

Core 2.3.0 covers everything merged since 2.2.0. It brings two larger pieces of work: search on Elasticsearch 8.18 with ES|QL, where index writes follow the database commit and repair themselves after an outage (https://github.com/zebbra/neops-core/pull/2184 and https://github.com/zebbra/neops-core/pull/2379), and centralized logging, where task logs go to a log store of your choice and carry the workflow run they belong to (https://github.com/zebbra/neops-core/pull/2409). Around them: locking by expansion for workflow acquires, safer host and rate-limit handling, and a series of fixes.

Take a database backup before you upgrade. The upgrade applies Core migrations up to 0204_merge_20261029_logging_develop, and the search indices are rebuilt.

At a glance

Area What changed Why it matters
Elasticsearch version Core needs Elasticsearch 8.18 or newer. Cross-entity search runs ES|QL LOOKUP JOIN (ES|QL is the query language cross-entity search uses), which older versions cannot parse.
Deploy The search indices are rebuilt automatically on upgrade: by ./neops up, and by Core itself in init.sh, which the Helm migrate hook runs (elastic_index --rebuild-if-stale). Search is unavailable or incomplete while the rebuild runs, and the Helm hook needs a longer --timeout.
Index sync Index writes run after the database commits, and every write carries the row’s version. A rolled-back change never reaches search, and an older write never replaces a newer one.
Restricted users A row whose access change has not reached the index yet is hidden from restricted users. Search never shows them what the database no longer grants.
Repair Rows whose index write failed are re-indexed on workflow lock requests, by the periodic index sync task where Celery beat runs (Helm: on by default; neops-docker-compose: the cms-tasks profile), or with manage.py resync_unsynced_rows. Without lock requests and without beat, search stays behind after an outage until you run the command.
Aggregations Aggregations over a related entity’s field (groups.name on devices, device.hostname on interfaces, interface.name on clients, …) return the buckets they returned before the indices were split. A field nothing has is an error. Dashboard tiles that went empty after the split show data again; a misspelled field says so instead of showing nothing.
Task log store NEOPS_LOG_BACKEND selects where task logs go: db, victorialogs (new) or loki, with NEOPS_LOG_URL for a remote store. Unset, Core writes them to stdout. Before, unset meant the database. A deployment that sets nothing can no longer read its task logs back: Query.logging answers queryable=false.
Log settings renamed NEOPS_LOGGER_* and EXECUTION_LOG_RETENTION_* are replaced by NEOPS_LOG_* names, and Core refuses to start while an old name is set. NEOPS_LOG_RETENTION takes a unit (30d, 12w). A deployment that still sets the old names does not start until they are renamed.
Workflow runs on log lines Lock and unlock lines carry the workflow run (workflow_execution_id), stored on LogEntry and written as a field to VictoriaLogs and Loki. Reading them needs workflow-execution-log:read. Core’s lines of a 2.0 workflow run can be found by its id, and execution:read, which every role holds, no longer shows them.
Provider logging Providers that support it have a per-task Log Level, NEOPS_TASK_LOG_LEVEL sets the default, and templates can call neops.log*, neops.fail and neops.skip. Bulky response bodies move to DEBUG, and a template can stop or skip an element with a reason instead of a syntax error.
Allowed hosts The image no longer sets DJANGO_ALLOWED_HOSTS=*. ALLOWED_HOSTS comes from NEOPS_PUBLIC_URL, or from DJANGO_ALLOWED_HOSTS when set, and the API refuses to start with neither. Host header validation is on by default.
Health probes /healthz (liveness) and /readyz (the database answers) are served before the host check; the chart probes them instead of /admin/login/. Probes keep working once real host names are configured.
Rate limits A rate-limited login, refreshToken, logout or exchangeOidcCode answers HTTP 429 with Retry-After. Clients and proxies can tell a throttle from a failed login, and know when to retry.
Metrics The neops_metrics plugin serves /metrics only with NEOPS_METRICS_TOKEN or NEOPS_METRICS_ALLOW_ANONYMOUS=true. Task names, fact keys and exception classes are no longer readable without a credential on the app port.
Locking Workflow acquires can expand to related entities (EntityAcquireInput.resolveRelated), and lock creation, release and the global lock switch are race-safe. Concurrent workflows no longer corrupt or lose locks.

Before upgrading, read “Before you upgrade”. After it, run the three checks in “After you upgrade, check”. To go back, follow “Going back to an earlier release”.

Upgrade notes

Before you upgrade

  • Both: Elasticsearch must be 8.18 or newer; see “Elasticsearch 8.18 or newer”.
  • Both: rename the log settings and choose a task log store; see “Task log settings”. Core does not start while an old name is set.
  • Both: set NEOPS_PUBLIC_URL or DJANGO_ALLOWED_HOSTS; see “Allowed hosts are required”. neops-docker-compose sets NEOPS_PUBLIC_URL from NEOPS_CMS_URL.
  • neops-docker-compose: ./neops up backs up only .env and its state file, not the database, so before git pull && ./neops up, take a backup with ./neops backup and note the neops-docker-compose commit you are on (git rev-parse HEAD; the repository has no release tags yet).
  • Helm: set --timeout and migrate.activeDeadlineSeconds as “Before helm upgrade” under “Where it runs” says. Note the Core image you upgrade to (quay.io/zebbra/neops-core:<the version>, or the image.tag you set, neops-core.image.tag under the umbrella chart; once it runs, kubectl -n <namespace> get deploy <release>-neops-core -o jsonpath='{.spec.template.spec.containers[0].image}' prints it), since a rollback by --atomic needs it, and the replica counts of the Core worker, beat and workflow engine Deployments (kubectl -n <namespace> get deploy), which going back restores.
  • Both: list the facts whose values disagree in type with the query in “Facts whose values disagree in type”, and fix them.

Task log settings

Core and the workflow engine read the same NEOPS_LOG_* names. Core refuses to start while one of the old names below is set, with an ImproperlyConfigured error that names its replacement:

Old name New name
NEOPS_LOGGER_CLASS NEOPS_LOG_BACKEND (db, victorialogs or loki), or NEOPS_LOG_CLASS for an explicit class
NEOPS_LOGGER_ENV NEOPS_LOG_ENV
NEOPS_LOGGER_LOKI_BASE_URL, NEOPS_LOGGER_VLOGS_BASE_URL NEOPS_LOG_URL
NEOPS_LOGGER_LOKI_USERNAME / _PASSWORD, NEOPS_LOGGER_VLOGS_USERNAME / _PASSWORD NEOPS_LOG_USERNAME / NEOPS_LOG_PASSWORD
NEOPS_LOGGER_VLOGS_INSERT_BATCH_SIZE NEOPS_LOG_BATCH_MAX_RECORDS
NEOPS_LOGGER_VLOGS_FLUSH_INTERVAL_SECONDS NEOPS_LOG_FLUSH_INTERVAL_MS (milliseconds)
EXECUTION_LOG_RETENTION_ENABLED NEOPS_LOG_RETENTION_ENABLED
EXECUTION_LOG_RETENTION_DAYS NEOPS_LOG_RETENTION, with a unit: 90d or 12w
EXECUTION_LOG_RETENTION_INTERVAL_MINUTES NEOPS_LOG_RETENTION_INTERVAL_MINUTES
EXECUTION_LOG_RETENTION_BATCH_SIZE NEOPS_LOG_RETENTION_BATCH_SIZE
  • Where task logs go. 2.2.0 wrote them to the database when nothing was set. 2.3.0 writes them to stdout when NEOPS_LOG_BACKEND is unset, where nothing can read them back. Set NEOPS_LOG_BACKEND=db to keep the 2.2.0 behaviour. victorialogs and loki also need NEOPS_LOG_URL; NEOPS_LOG_TENANT sets the Loki X-Scope-OrgID or the VictoriaLogs <account>[:<project>]. neops-docker-compose sets db by default, and victorialogs with its compose.logs.yaml overlay. The core chart sets db in its values.yaml.
  • Retention. NEOPS_LOG_RETENTION defaults to 30d, where EXECUTION_LOG_RETENTION_DAYS defaulted to 90 days; the core chart ships 90d. A bare number refuses to start, since it means days to Core and months to VictoriaLogs. Retention is still off unless NEOPS_LOG_RETENTION_ENABLED is true, and it deletes executions older than the horizon with their results and logs, not only the logs.
  • Global task log level. NEOPS_TASK_LOG_LEVEL (DEBUG or INFO, default INFO) applies to tasks without their own Log Level. WARNING refuses to start, since providers that do not classify their lines log everything at INFO.
  • Old logs stay where they are. Switching the store does not move existing lines; manage.py migrate_logs copies them between the database, Loki and VictoriaLogs.

Allowed hosts are required

The image used to set DJANGO_ALLOWED_HOSTS=*, which turned off Django’s host check and let a request’s Host header decide the OIDC login URLs Core hands out. That default is gone. ALLOWED_HOSTS is now DJANGO_ALLOWED_HOSTS when it is set (including *), otherwise the host of NEOPS_PUBLIC_URL. With neither, the API refuses to start (ImproperlyConfigured: No allowed Host is configured); management commands and Celery are not affected. When other services reach the API by another name, such as the in-cluster Service, list every name in DJANGO_ALLOWED_HOSTS, the public host included. /healthz and /readyz answer any host, so point probes there; the core chart does.

Metrics endpoint needs a credential

With the neops_metrics plugin enabled, /metrics is mounted only when NEOPS_METRICS_TOKEN is set (scrape with that bearer token) or NEOPS_METRICS_ALLOW_ANONYMOUS=true. Without either, the route is not mounted and a scraper that read it before gets nothing. The standalone metrics exporter still has no authentication; keep its port private.

Elasticsearch 8.18 or newer

sync_lookup_indices, which elastic_index --rebuild runs, fails on an older Elasticsearch. Upgrade Elasticsearch before Core.

The rebuild runs on upgrade, and search is down while it runs

The index definitions change in this release, so the upgrade rebuilds the indices. The rebuild deletes and recreates them, then fills them from the database. Until it finishes:

  • search returns no or partial results;
  • a search that finds no index fails with “Search is not available right now: an index it needs is missing or being rebuilt” instead of rebuilding it inside the request;
  • workflow locks that resolve their entities through search are refused with INDEX_SYNC_PENDING, which the engine retries. A rebuild that fails or is killed part way keeps them refused, with a message naming it, until a rebuild succeeds; each new rebuild takes over the one before it (it marks the earlier, stopped one finished and redoes its work).

If Elasticsearch refuses documents, the rebuild finishes anyway, exits with code 3, and lists them: fix them as the next paragraph describes, and list them again with manage.py index_sync_status --list rejected --limit 0. init.sh takes code 3 as a warning and goes on. --rebuild and --populate set refused documents aside the same way. The usual cause is a fact whose values disagree in type between rows: the rebuild gives a field the type of the first value it indexes, and refuses the rows whose value does not fit it. Each refused row is set aside: it is marked rejected, stays out of search, and is indexed again once it is saved with data Elasticsearch accepts. Until then it is also out of reach of the workflow locks that resolve entities through search: such a lock is granted without the row, and says nothing about it. With https://github.com/zebbra/neops-docker-compose/pull/25, ./neops doctor warns while any remain. The rebuild finishes all the same, and searching locks are accepted again.

The first value the rebuild indexes sets the field’s type, so a few odd rows can get many others refused, and saving a refused row cannot get it in while the field keeps the other type. The rebuild indexes in a fixed order, so the same database gets the same documents refused on every rebuild: index by index (clients, executions, groups, network), and in network all devices before any interface, so a key that devices and interfaces both carry takes its type from the devices. Within a model the rows go in its own order, ties by id: devices by hostname, interfaces by name, groups by name descending, clients and executions by id. With --parallel (ELASTICSEARCH_DSL_PARALLEL) or more than one shard (ELASTICSEARCH_DSL_INDEX_SHARDS), Elasticsearch can see the documents in another order. Names sort by the database’s collation, so a restore onto a Postgres with another collation can change which rows are refused. Core already has the means to keep a fact or check to one type: its schema in /admin/core/elasticsyncconfig/ (“Generate schemas”, or edit the key’s schema), which the index respects; values that do not fit it are indexed as null. For a type conflict, give the key one type there, or turn its sync off, or change the rows of the other type; “After upgrading: give every flagged fact and check key one type” below finds the keys. Then run elastic_index --rebuild --force, since --rebuild-if-stale finds the index current. A refusal that reads “Limit of total fields [N] has been exceeded” means the index holds as many fields as ELASTICSEARCH_DSL_INDEX_FIELDS_LIMIT allows. Restrict the synced keys with schemas, or turn sync off for noisy keys, then run elastic_index --rebuild --force; or raise that setting, which changes the schema hash, so the next --rebuild-if-stale rebuilds.

The report: before exiting, the rebuild prints “rebuilt; N documents refused and set aside” with the count per model, then the refusals grouped by the field Elasticsearch names (for example “facts.kind: 10,004 rows, type conflict: …”), each with its remedy, then each refused document with Elasticsearch’s reason, up to 50, and the command that lists them all. That command does not list executions yet; the rebuild’s output does, and so does SELECT id FROM core_execution WHERE elastic_sync_status = 'rejected'. The report goes to stdout, and the exit message, “DocumentsSetAside: rebuilt; N documents refused and set aside, listed above”, to stderr, so in container logs that line can land inside the report rather than after it.

Core decides on its own whether to rebuild. elastic_index --rebuild-if-stale rebuilds each index that this release or this Elasticsearch version did not build, and returns at once when there is none. A rebuild that finishes, --rebuild-if-stale or --rebuild, records in each index it rebuilt, under _meta.neops_build in the mapping (its build record): the schema hash (a fingerprint of the index definitions), the Elasticsearch version, and when the database applied Core’s migration 0203_index_stamp_epoch. An index without that record is stale: one that --create just made, one that a killed or failed rebuild left behind, and every index built before this release. So is every index recorded before a rollback that migrated the database back past 0203_index_stamp_epoch: the older Core wrote rows without keeping the indices up to date, and the upgrade after it applies the migration again, at a new time. The Helm rollback steps below undo the migrations, as compose’s do. A rollback that leaves them applied does not reset the record; where the older Core rebuilt groups or clients, their missing record still gets network rebuilt, see below. A rebuild that set documents aside records it too, so a later --rebuild-if-stale does not retry them.

  • A later release that changes one index rebuilds only that index. A rebuilt network, groups or clients index brings the four lookup indices (four small indices that link devices, groups, interfaces and clients for cross-entity search) with it, and a changed lookup index brings network, groups and clients. So does a network, groups or clients index that is missing or has no build record, since whatever wrote it may have changed group memberships that the other two still hold; one whose record names only another schema hash is rebuilt without them.
  • The first run after an Elasticsearch upgrade rebuilds every index, even for a patch version; with Helm that is the next install or upgrade of Core. So does a change to ELASTICSEARCH_DSL_INDEX_SHARDS, ELASTICSEARCH_DSL_INDEX_REPLICAS or ELASTICSEARCH_DSL_INDEX_FIELDS_LIMIT.
  • On Elasticsearch older than 8.18, --create and --rebuild-if-stale fail before they replace or delete any index.
  • When two callers run it at once, the second waits for the first, then finds nothing to do. elastic_index --create, --delete, --populate and --rebuild wait the same way, and so does sync_lookup_indices. A waiting command prints “Waiting for another rebuild to finish”; under init.sh the log pauses after “Setup elastic indexes (waits while another rebuild runs)”.
  • On an empty database it takes a few seconds.

Where it runs:

  • Helm. The core chart’s migrate Job, a post-install,post-upgrade hook, runs init.sh from the Core image it deploys. init.sh now runs elastic_index --rebuild-if-stale after --create and fails when it fails. When the rebuild only set documents aside, init.sh logs “Rebuild stale elastic indexes: Warning: Elasticsearch refused documents, which are set aside (see above)” and the hook succeeds. The new Core pods already serve requests while the hook runs, since their readiness check only asks the database; search is down from the start of the rebuild until the hook finishes.
  • Before helm upgrade: Kubernetes stops the Job after migrate.activeDeadlineSeconds (default 900 seconds), and Helm waits for the hook only until --timeout (default 5m). For this upgrade, and for the first upgrade after an Elasticsearch upgrade, set the deadline above the time a rebuild takes on your inventory and --timeout a few minutes above the deadline, since the deadline also covers the image pull, the waits and the migrations. With the neops umbrella chart, for example: helm upgrade ... --timeout 35m --set neops-core.migrate.activeDeadlineSeconds=1800. As a reference, 5,000 devices, 50,000 interfaces and 5,000 clients without facts rebuilt in 30 to 50 seconds on a laptop; facts and checks make every document larger.
  • During: in a second terminal, as soon as kubectl -n <namespace> get job <release>-neops-core-migrate lists the Job (neops-neops-core-migrate for a release named neops), keep its log: kubectl -n <namespace> logs -f job/<release>-neops-core-migrate | tee migrate.log. Helm deletes the Job, and its log, once it succeeds, and that log is the only one that gives Elasticsearch’s reason for each document set aside.
  • After: list the rows the rebuild set aside: kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py index_sync_status --list rejected --limit 0. It exits 0 whatever it lists. To gate a deployment, run index_sync_status without --list: it exits 1 while any row is rejected, and also while a rebuild is unfinished, unlocks are quarantined, rows have been pending or in error too long, or group writes or unlocks have waited too long for their index writes (--max-pending-age sets how long, except for rows in error). A later run that finds nothing to rebuild prints only how many rows are still set aside. elastic_index --populate --no-sync-lookups --models <app.model> prints the reasons again; without --no-sync-lookups it also recreates all four lookup indices.
  • After, if migrate.log has no “Rebuild stale elastic indexes:” line: the hook ran an init.sh without this step. Every core chart on oci://quay.io/zebbra/charts (2.1.2 and later) runs init.sh from the image it deploys, so the pods most likely run a Core image from before this release; such an image has no --rebuild-if-stale. Check which image they run: kubectl -n <namespace> get deploy <release>-neops-core -o jsonpath='{.spec.template.spec.containers[0].image}'. By default it is the core chart’s appVersion; under the neops umbrella chart, that is the neops-core version neops-helm’s charts/neops/Chart.yaml pins. image.tag (neops-core.image.tag under the umbrella chart) overrides it. If it is not this release’s image, set that tag to this release’s version and run helm upgrade again. If it is, run the rebuild yourself: kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild-if-stale. Without a migrate.log, check the image the same way, then run it: it returns at once when every index is current. Exit 0 means done. Exit 3 means it finished but set rows aside; list them as above. Any other exit means it failed: fix the cause and run it again. Search is down while it runs, and the command stops if your terminal session drops.
  • Helm gave up (“timed out waiting for the condition” in Helm 3, “context deadline exceeded” in Helm 4) while kubectl -n <namespace> get job <release>-neops-core-migrate still shows the Job active: it keeps rebuilding. Do not run helm upgrade again yet: it deletes the running Job and the new one starts the rebuild over. Wait for it with kubectl -n <namespace> wait --for=condition=complete job/<release>-neops-core-migrate --timeout=30m (the deadline you set); the next helm upgrade then finds nothing to rebuild. --atomic (--rollback-on-failure in Helm 4) rolls Core back to the previous release while the Job goes on rebuilding the indices for the new one, so leave it off for this upgrade or give it a long enough --timeout.
  • kubectl -n <namespace> get job <release>-neops-core-migrate shows the Job failed (Helm 3 says “DeadlineExceeded” or “BackoffLimitExceeded”, Helm 4 “status: Failed”): search stays empty or partial, and lock requests that search stay refused, until a rebuild succeeds. Kubernetes retries a failed rebuild from the start up to migrate.backoffLimit (10) times; the same error twice in the log will not heal on its own. Fix the cause, then run helm upgrade again with a longer deadline and --timeout. From a running Core pod, kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild-if-stale does the same, but stops if your terminal session drops.
  • index_sync_status --list rejected --limit 0 lists rows: they stay out of search until they are fixed as described above. The query in the section on facts whose values disagree in type finds such rows before an upgrade.
  • The log stops at “Waiting for another rebuild to finish” or after “Setup elastic indexes (waits while another rebuild runs)”: another rebuild holds the lock. Find it with SELECT a.pid, a.client_addr, a.backend_start FROM pg_locks l JOIN pg_stat_activity a USING (pid) WHERE l.locktype = 'advisory' AND l.classid = 1852141424 AND l.objid = 1932355960 AND l.granted AND l.database = (SELECT oid FROM pg_database WHERE datname = current_database()); in the Core database. A session whose pod is gone is ended with SELECT pg_terminate_backend(<pid>);. The Core image has no psql; to run a query, put it in query.sql and pass it to Core’s Django shell: kubectl -n <namespace> exec -i deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py shell -c "import sys; from django.db import connection; cursor = connection.cursor(); cursor.execute(sys.stdin.read()); print(cursor.fetchall())" < query.sql. With neops-docker-compose, ./neops compose -- exec -T postgres-cms psql -U neops -d neops opens the Core database.
  • Going back to an earlier release: see “Going back to an earlier release” below.
  • The names in these notes assume a release name without neops-core in it and no fullnameOverride; kubectl -n <namespace> get job,deploy -l app.kubernetes.io/name=neops-core lists the real ones.
  • neops-docker-compose. With https://github.com/zebbra/neops-docker-compose/pull/27, every ./neops up and ./neops install runs elastic_index --rebuild-if-stale in the cms container once it is healthy; its cms-init service does not run init.sh. Exit 3 is a warning: up logs that documents were set aside, the number of rows marked rejected and the command that lists them, and records the start (the images it started, which the next up compares against). Any other non-zero exit fails up, which then does not record the start. For a Core without --rebuild-if-stale, up runs elastic_index --rebuild --force whenever the Core or the Elasticsearch image changed, and fails on any non-zero exit, 3 included. A compose release without that PR still decides from elastic_index --schema-hash, fails on exit 3, and so rebuilds again on every ./neops up until no row is refused.
  • Neither. Run the same steps as init.sh after each upgrade:
cd backend
uv run python manage.py migrate
uv run python manage.py elastic_index --create
uv run python manage.py elastic_index --rebuild-if-stale

Exit code 3 from the last command means it finished but set documents aside; uv run python manage.py index_sync_status --list rejected --limit 0 lists them. A script under set -e or a configuration tool stops on 3 unless it accepts that code, as init.sh does.

A manual elastic_index --rebuild records the indices it rebuilt, also when it set documents aside, so the next --rebuild-if-stale leaves them alone. --rebuild --models records only the indices of those models, --rebuild --no-sync-lookups leaves the lookup indices unrecorded, and --populate records nothing.

This release keeps devices and interfaces in one index, network. The devices and interfaces indices of 2.2.0 and older stay behind after the upgrade and are no longer written. Delete them with ./neops compose -- exec -T cms python manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['devices', 'interfaces'], ignore_unavailable=True)" (Helm: the same manage.py shell command through kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py). A rollback to such a release recreates them with the elastic_index --rebuild --force its steps run.

The rebuild is required, not only recommended. Documents written before this release carry Elasticsearch’s own version numbers, which are higher than the new row versions. A device, interface, client or group document moves its row past such a number on its next write, and so do the lookup rows that cross-entity search joins on; only the rows nobody writes stay stale until the rebuild recreates them.

Facts whose values disagree in type

Devices and interfaces now share one mapping in the network index. Elasticsearch refuses a device or interface whose fact or check does not fit the type an earlier device or interface gave the same field: a number against a word (mtu: 1500 on a device, mtu: "jumbo" on an interface), a date against a word ("2026-10-01T10:00:00Z" against "never", or against "2026-10-01 10:00:00", which Elasticsearch keeps as text), two date forms ("2026/10/01" against "2026-10-01"), or an object against a plain value. Two devices, or two interfaces, that disagree were refused before this release too. The rebuild sets the refused rows aside and finishes, as described above: they stay out of search, and out of every dashboard tile, until they are fixed.

An object against a plain value, true/false against a number or a date, two date forms, or a conflict in both directions fail every time. A word against a number, true/false or a date, and a date against a number, fail only when the word or the date is indexed after the other value. Devices are indexed before interfaces, so a key that holds a number on a device and a word or a date on an interface is refused on the interfaces; with --parallel or more than one shard the order, and so the rows set aside, can change from run to run. Retrying is no fix: fix every field the query below lists first.

Before upgrading (compose: before git pull), list the candidates. The query is read-only. It walks the facts and checks of every device and interface that is not deleted down to their values, inside objects and arrays, and lists each field whose values are of more than one kind: number, boolean, string, date, date yyyy/MM/dd (a date written 2026/10/01) or object. It lists more than Elasticsearch refuses: Elasticsearch accepts "1500" where it expects a number. It misses a few rare cases:

  • Neops turns an object whose values are all objects, the first two with a key in common (the facts object itself included), into a list, with each outer key as id, so those values reach Elasticsearch under other fields;
  • Elasticsearch splits a key that contains a dot ("vlan.100") into nested fields;
  • it reads a few strings differently from Elasticsearch: a time zone given by name (2026-10-01T10:00:00Europe/Zurich), a negative year, a day that does not exist (2026-02-30, 2026/04/31), year 0 written 0000/01/01, and a time after a date with no day (2026T10, 2026-02T10:20), which Elasticsearch 8 reads as a date (2026T10 as 1970-01-01 10:00) and the query, like Elasticsearch 9, counts as a string;
  • Elasticsearch stores a value named ip, ip4, ip6, ipv4, ipv6 or ipaddress (in any case, with an optional leading or trailing _ or -) as an address, and refuses one that is not an address (IPv4: "dhcp") on its own, as it did before this release. Neops drops such values first only in facts whose name contains ip in lower case (ipv4, ipAddress); it keeps them in other facts (IP, IPv4) and in checks.

An empty result therefore makes a refusal unlikely, not impossible; the rebuild names any row it sets aside.

The query reads one row at a time. Its temporary files on the CMS database’s volume are freed row by row and stay at up to about ten times the largest row’s facts, more when those nest deeply, unless facts hold maps keyed by values rather than fixed names, such as BGP peers keyed by their address or routes keyed by prefix: on 5,000 devices with 60 peers each, keyed by each device’s own peer addresses, and 100,000 interfaces (150 MB of facts and checks) they reached 1.1 GB and it took 70 s; with the same peers on every device it wrote none and took 27 s. Its first line stops it once its temporary files reach 2 GB, since by default that volume also holds Elasticsearch’s data. If it stops with “temporary file size exceeds temp_file_limit”, raise the limit to less than the free space ./neops compose -- exec -T postgres-cms df -h /var/lib/postgresql/data shows, minus 5% of the volume’s size: Elasticsearch blocks writes to every index when its disk is 95% full. Setting the limit needs a database superuser, which compose’s neops user is.

./neops compose -- exec -T postgres-cms psql -v ON_ERROR_STOP=1 -U neops -d neops -f - <<'SQL'
SET temp_file_limit = '2GB';
WITH source(entity, id, field, value) AS (
  SELECT 'device', id, 'facts', facts FROM core_device WHERE deleted_at IS NULL
  UNION ALL SELECT 'device', id, 'checks', checks FROM core_device WHERE deleted_at IS NULL
  UNION ALL SELECT 'interface', id, 'facts', facts FROM core_interface WHERE deleted_at IS NULL
  UNION ALL SELECT 'interface', id, 'checks', checks FROM core_interface WHERE deleted_at IS NULL
), kinds AS (
  SELECT s.entity, k.path, k.kind, count(*) AS rows, min(s.id) AS example
  FROM source s CROSS JOIN LATERAL (
    WITH RECURSIVE node(path, value) AS (
      SELECT s.field, s.value
      UNION ALL
      SELECT n.path || coalesce('.' || c.key, ''), c.value
      FROM node n CROSS JOIN LATERAL (
        SELECT key, value FROM jsonb_each(CASE WHEN jsonb_typeof(n.value) = 'object' THEN n.value ELSE '{}' END)
        UNION ALL
        SELECT NULL, value FROM jsonb_array_elements(CASE WHEN jsonb_typeof(n.value) = 'array' THEN n.value ELSE '[]' END)
      ) AS c
    )
    SELECT DISTINCT path,
      CASE
        WHEN jsonb_typeof(value) <> 'string' THEN jsonb_typeof(value)
        WHEN value #>> '{}' ~ '^[0-9]{4}(-(0[1-9]|1[0-2])(-(0[1-9]|[12][0-9]|3[01])T?)?)?$' THEN 'date'
        WHEN value #>> '{}' ~ '^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3])(:[0-5][0-9](:[0-5][0-9]([.,][0-9]{1,9})?)?)?(Z|[+-]((0[0-9]|1[0-7])(:[0-5][0-9](:[0-5][0-9])?|[0-5][0-9]([0-5][0-9])?)?|18(:00(:00)?|00(00)?)?))?$' THEN 'date'
        WHEN value #>> '{}' ~ '^[0-9]{4}/(0[1-9]|1[0-2])/(0[1-9]|[12][0-9]|3[01])( ([01][0-9]|2[0-3]):[0-5][0-9]:[0-5][0-9])?$' THEN 'date yyyy/MM/dd'
        ELSE 'string'
      END AS kind
    FROM node WHERE jsonb_typeof(value) NOT IN ('array', 'null')
  ) AS k
  GROUP BY s.entity, k.path, k.kind
)
SELECT path AS field, string_agg(format('%s on %ss (rows: %s, e.g. id %s)', kind, entity, rows, example), '; ' ORDER BY kind, entity) AS kinds
FROM kinds GROUP BY path HAVING count(DISTINCT kind) > 1
ORDER BY 1;
SQL

Each row names a field and, for each kind of its values on devices and on interfaces, how many rows hold it and the id of one of them. Make the values agree, or remove the odd ones, before the upgrade; a row set aside after it is indexed again once its values agree, or its key has one type as the next section describes, and the indices are rebuilt. Facts usually come from a workflow, so fix what writes them too, or its next run brings the conflict back. Without neops-docker-compose, run the query with psql -v ON_ERROR_STOP=1 against the CMS database; as a user who is not a superuser, psql stops at the first line with “permission denied to set parameter”, so remove that line and watch the free space instead. The same refusal hits a device or interface written after the upgrade; “Aggregations over related entities” below says what it does then.

After upgrading: give every flagged fact and check key one type

  • When: the upgrade’s rebuild exited 3 or logged “rebuilt; N documents refused and set aside” (under init.sh: “Rebuild stale elastic indexes: Warning: Elasticsearch refused documents …”), or manage.py index_sync_status --list rejected --limit 0 lists rows. Running the analysis after every upgrade does no harm. The check exists only from this release on, so it cannot run before the upgrade.
  • Find the keys: run Generate schemas in /admin/core/elasticsyncconfig/ (a dry run is enough), or the command. --sample-size -1 reads every row; the command’s default reads 100 per entity type and can miss the rows that conflict.
# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_sync_schema --analyze --sample-size -1
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_sync_schema --analyze --sample-size -1

It lists under “Type conflicts” every fact and check key whose schemas let values of more than one Elasticsearch type into one field: one schema allowing both an integer and a string, for example, or a device and an interface schema typing the same key differently, since devices and interfaces share one index. It reads the schemas only, so an empty list is not a guarantee: a key no schema covers (none stored, and none generated in the run), a property its schema does not name, and facts whose dict of dicts is indexed as one list (NEOPS_ELASTIC_CONVERT_FACTS, on by default) can still mix types. - Fix them: in /admin/core/elasticsyncconfig/, find the row by its fact key, entity type and data type. A flag labelled “(generated)” has no row yet, which is the case for every key of an install that synced without schemas: run Generate schemas with Create new, or the command with --create --sample-size -1 in place of --analyze --sample-size -1, then edit the row it created. Create new stores a schema built from today’s values for every key that has none, not only the flagged ones, so from then on a value of another type indexes as null instead of being refused. To cover only the flagged key, add its row by hand in the same admin page. Give the flagged path one type, for example "type": "string" in place of "type": ["integer", "string"], or keep one branch of an anyOf; for a device and interface disagreement, make both rows agree. A row whose value no longer fits then indexes the whole key, or a check’s whole metrics, as null. Or turn the row’s sync off, which stops indexing the key. Then rebuild. Like the upgrade’s rebuild, it deletes and recreates the indices, so search is incomplete until it finishes:

# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_index --rebuild --force
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild --force

The rebuild logs a “Schema validation failed” warning for each key and entity type whose rows were indexed as null, at most one a minute, with the number of rows.

The fix worked when this rebuild exits 0, not 3, and manage.py index_sync_status --list rejected --limit 0 lists nothing. Rows still listed were refused for a reason the analysis cannot see; what it reads is described under “Find the keys”. - Update saved queries: once a key is typed as a number, date, boolean or IP address, a query for a value of the old type fails instead of matching nothing. For example, facts.mtu:jumbo on an mtu now typed integer answers “Query failed: For input string”, and a dashboard tile saved with it shows that error. Change or remove those queries.

Locations’ facts and checks are now filtered by the device_group schemas when they are saved, as the rebuild already did, so each location re-syncs once.

Stored aggregations: id fields renamed, fields that do not aggregate reported

Migration 0202_scope_aggregation_id_fields stores device.id (interface aggregations), interface.id and location.id (client aggregations) as device_id, interface_id and location_id, the fields that hold the ids now. Both spellings keep working in queries and in dashboard cards.

An aggregation whose field no entity has (a typo, or a path that never aggregated) now fails with an error that names the accepted paths, instead of showing no buckets. In the web client the card stays in its loading state and a short “Error loading” notice appears. After upgrading, list them:

./neops compose -- exec -T cms python manage.py check_aggregation_fields   # neops-docker-compose
uv run python manage.py check_aggregation_fields                        # source checkout, in backend/

It reads the scope aggregations and the aggregation cards of every scope’s dashboards (the overview dashboard and the four entity detail dashboards), and changes nothing. For each one it lists, it says where it is, why its field does not aggregate, which paths the entity accepts, and how to fix it. The web client edits neither, so:

  • a dashboard card: send the scope’s corrected JSON with the scopesUpsert mutation, or edit it in the CMS admin site (<NEOPS_CMS_URL>/admin/, Scopes, the scope, the JSON field and path the check prints), or remove the card. The admin site refuses to save a scope whose overview dashboard (dashboard_configuration) is still empty ({}), so use the mutation for those;
  • a scope aggregation: edit it in the CMS admin site (Scope aggregations) or with the scopeAggregationUpsert mutation, or delete it.

It exits 0 when every field aggregates, 2 when it lists some, and 1 when it cannot read the database, or part of it; it still lists what it could read. With --json the last line of its output is the result as JSON: {"unsupported": [...]}, plus "error" when it could not read everything.

manage.py migrate prints the same findings as warning core.W001 before it migrates; under ./neops up they appear in ./neops compose logs cms-init. When upgrading from a release before 2.1, the database has no dashboard columns yet at that point: the scope aggregations are still listed, and warning core.W002 says the cards could not be read, so run the command after the upgrade for them. A new installation, which has no tables yet, prints neither.

After you upgrade, check

Run each with ./neops compose -- exec -T cms python manage.py <command> (Helm: kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py <command>):

  • index_sync_status --list rejected --limit 0 lists nothing; rows it lists were set aside, see “After upgrading: give every flagged fact and check key one type”.
  • check_aggregation_fields exits 0; exit 2 lists aggregations to fix, see “Stored aggregations: id fields renamed, fields that do not aggregate reported”.
  • elastic_sync_schema --analyze --sample-size -1 lists no “Type conflicts”.

Going back to an earlier release

Going back undoes the logging migration too: migrate core 0188_refreshtoken_auth_provider removes 0192_logentry_workflow_execution_id, so log lines lose the workflow run they were linked to. Restoring the backup you took before the upgrade is the only way to get the database exactly as it was; everything written since is then lost. The earlier release reads the old log setting names: put NEOPS_LOGGER_* and EXECUTION_LOG_RETENTION_* back where you set NEOPS_LOG_* yourself (Helm values, an .env of your own), or it falls back to its defaults (database logs, retention off), and remove them again before the next upgrade. Keep DJANGO_ALLOWED_HOSTS or NEOPS_PUBLIC_URL; the earlier image ignores neither.

First, on both, while this release still runs

Make every device and interface fit the types its facts and checks are indexed with. Otherwise the earlier release’s rebuild (with neops-docker-compose, the one ./neops up --allow-downgrade runs) fails with “BulkIndexError: document(s) failed to index.”, search stays empty, and saving the affected devices, their interfaces and their groups’ memberships then fails too. List the keys and keep the list, since the earlier release cannot list the conflicts itself:

# neops-docker-compose
./neops compose -- exec -T cms python manage.py elastic_sync_schema --analyze --sample-size -1 > type-conflicts-before-rollback.txt
# Helm
kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_sync_schema --analyze --sample-size -1 > type-conflicts-before-rollback.txt

Then give each flagged key one type in /admin/core/elasticsyncconfig/, or turn its sync off; or fix the rows. Then export the stored schemas, with the changes you just made:

# neops-docker-compose
./neops compose -- exec -T cms python manage.py dumpdata core.ElasticSyncConfig --indent 2 --output /tmp/elastic-sync-config.json
./neops compose -- cp cms:/tmp/elastic-sync-config.json ./elastic-sync-config.json
# Helm (not rehearsed)
POD=$(kubectl -n <namespace> get pod -l app.kubernetes.io/name=neops-core,app.kubernetes.io/instance=<release>,app.kubernetes.io/component=app -o name | head -1)
kubectl -n <namespace> exec "$POD" -- /docker-entrypoint.sh ./manage.py dumpdata core.ElasticSyncConfig --indent 2 --output /tmp/elastic-sync-config.json
kubectl -n <namespace> cp "${POD#pod/}":/tmp/elastic-sync-config.json ./elastic-sync-config.json

--output is needed: redirecting stdout puts Core’s start-up output into the file. A restored backup holds the schemas from before the upgrade, and 2.2.0 cannot list the conflicts, so the way back from the backup loads this export (step 3 of “neops-docker-compose, from the backup”). On the ways that keep the data (“neops-docker-compose, keeping the data written since”, and “Planned.” under “Helm”), the rows survive migrate core 0188_refreshtoken_auth_provider, so the export is a safety copy there. The Helm commands take one pod, so that the export and the copy run in the same one. index_sync_status --list rejected --limit 0 lists the rows this release set aside. The earlier release filters facts and checks through the same stored schemas when it indexes them (in 2.2.0, ElasticSyncConfig), so its rebuild then succeeds. On Helm, the way back is the one under “Helm” below; going back by restoring a database backup is not covered here.

neops-docker-compose, keeping the data written since

To keep the data written since, undo this release’s migrations while its image is still in place, with nothing writing, then start the earlier release, which rebuilds its search indices, and delete the ones only this release uses. If the commit you noted pins an older Elasticsearch, empty its data directory first (the block’s find line does it only then), as in step 2 of “neops-docker-compose, from the backup” below; set NEOPS_DATA_DIR in the shell as there.

./neops compose -- stop cms cms-worker cms-beat engine
./neops compose -- run --rm --no-deps cms python manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145
./neops compose -- run --rm --no-deps cms python manage.py migrate core 0188_refreshtoken_auth_provider   # the last core migration of 2.2.0
./neops compose -- stop elasticsearch
current=$(git show HEAD:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
pinned=$(git show <the commit you noted>:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
echo "leaving Elasticsearch ${current:-unknown}, going back to ${pinned:-unknown}; an older or unknown pin empties the data"
[ -n "$current" ] && printf '%s\n' "$current" "$pinned" | sort -V -C || sudo find "${NEOPS_DATA_DIR:-./data}/elasticsearch" -mindepth 1 -delete   # only when that commit pins an older Elasticsearch
git checkout <the commit you noted>
./neops up --allow-downgrade
./neops compose -- exec -T cms python manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['network', 'device_group_edges_by_device_lookup', 'device_group_edges_by_group_lookup', 'client_group_edges_lookup', 'client_interface_edges_lookup'], ignore_unavailable=True)"

The last command deletes the network index and the four lookup indices: the earlier release neither uses nor rebuilds them, so they would keep this release’s documents, which go stale, and the next upgrade must rebuild them. After going back to 2.2.1-beta.2 to 2.2.1-beta.5, which already use them, skip it. After going back to 2.2.1-beta.7, run it: that beta does not use them. Do not run the earlier release’s elastic_index --rebuild on top of the one ./neops up ran: 2.2.0’s stops at the first document Elasticsearch refuses and leaves search empty. The cron migration removes the “Unlock Index Sync” periodic task this release added, which 2.2.0 has no task for; skip it on an image without the cron plugin (it answers “No installed app with label ‘cron’”). For another release, python manage.py showmigrations core cron in its image names the last migrations. migrate core 0188_refreshtoken_auth_provider leaves four migrations 2.2.0 does not have applied, since 0188 does not depend on them: 0175_add_elastic_relation_hash_fields and the merges 0181_merge_20260728_0703, 0183_merge_20260810_2137 and 0185_merge_20260917_esql. The columns 0175 added stay on the device, interface, group, client and execution tables: elastic_relation_hash (not null, empty by default) and elastic_relation_synced_at (nullable). 2.2.0 does not read them and inserts rows without them, and the next upgrade finds the four applied; no migration target undoes only them. Undoing 0202_scope_aggregation_id_fields stores device.id, interface.id and location.id again; a dashboard card saved with device_id, interface_id or location_id after the upgrade keeps that field, which 2.2.0 does not aggregate.

neops-docker-compose, from the backup

Going back with the backup from “Before you upgrade” takes four steps. Their commands read NEOPS_DATA_DIR from the shell, so if .env sets it, export the same value first:

  1. Stop the stack, note the Elasticsearch this release pins, and check out the commit you noted, in the shell you use for the next steps:

    ./neops down
    current=$(git show HEAD:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
    git checkout <the commit you noted>
    
  2. If that commit pins an older Elasticsearch than the one you leave, empty its data directory and keep the directory itself, which Elasticsearch (uid 1000) owns: Elasticsearch cannot open data a newer version wrote, and the rebuild in step 4 refills it. Compose commits before 2026-10-27 pin 8.9.2.

    pinned=$(git show HEAD:compose.yaml | grep -o 'elasticsearch/elasticsearch:[0-9.]*' | cut -d: -f2)
    echo "leaving Elasticsearch ${current:-unknown}, going back to ${pinned:-unknown}; an older or unknown pin empties the data"
    [ -n "$current" ] && printf '%s\n' "$current" "$pinned" | sort -V -C || sudo find "${NEOPS_DATA_DIR:-./data}/elasticsearch" -mindepth 1 -delete   # only when it pins an older Elasticsearch
    
  3. Move the database directories aside, so the restore loads into empty databases: it drops only the tables the backup holds, and the tables this release added would make the next upgrade’s migration fail.

    sudo mv "${NEOPS_DATA_DIR:-./data}/cms/postgres" "${NEOPS_DATA_DIR:-./data}/cms/postgres.after-upgrade"
    sudo mv "${NEOPS_DATA_DIR:-./data}/engine/postgres" "${NEOPS_DATA_DIR:-./data}/engine/postgres.after-upgrade"   # only if that commit pins another engine
    

    Then run steps 1 and 2 of “Restore” in neops-docker-compose’s docs/30-operations.md (put .env back, start the databases, pg_restore) and stop there: its step 3 deletes the search indices and then runs ./neops up without --allow-downgrade; here step 4’s up --allow-downgrade rebuilds the indices instead. That step 2 restores both dumps, engine.dump too, so the workflow engine goes back to the same point as Core: the workflow data written since the upgrade is lost too.

    Then, while only postgres-cms and postgres-engine run, load the sync schemas you exported under “First, on both, while this release still runs”:

    ./neops compose -- run --rm --no-deps -T cms python manage.py loaddata --format=json - < elastic-sync-config.json
    

    It reads the file on stdin, since a run --rm container cannot receive a cp. loaddata keeps the dump’s ids, so the restored table must not already hold rows with the same fact key, entity type and data type under other ids (2.2.0’s unique_together); a 2.2.0 install that never used Generate schemas has none. If it has, the load fails with django.db.utils.IntegrityError: Problem installing fixture '-': Could not load core.ElasticSyncConfig(pk=<id>): duplicate key value violates unique constraint "core_elasticsyncconfig_fact_key_entity_type_dat_931cc391_uniq" and loads nothing, since loaddata runs in one transaction. Then delete every stored schema row, which the export file replaces, and load again, still with only the databases running:

    ./neops compose -- run --rm --no-deps -T cms python manage.py shell -c "from neops.core.models import ElasticSyncConfig; print(ElasticSyncConfig.objects.all().delete())"
    ./neops compose -- run --rm --no-deps -T cms python manage.py loaddata --format=json - < elastic-sync-config.json
    
  4. Run ./neops up --allow-downgrade, which rebuilds the earlier release’s search indices since the Core image changed, then delete the ones only this release uses with the last command of “neops-docker-compose, keeping the data written since” above. If up still fails with “document(s) failed to index”, fix the key in 2.2.0’s /admin/core/elasticsyncconfig/, using the type-conflicts-before-rollback.txt you made under “First, on both, while this release still runs”, and run ./neops up --allow-downgrade again. It worked when up exits 0; ./neops doctor alone is no proof, since it passed with an empty groups index after a failed up.

Switch back to your branch before the next git pull.

Helm

Planned. helm rollback does not undo this release’s migrations, and the earlier release cannot run on them: it fails to create devices, interfaces, clients, groups and executions, since their new columns have no database default, and it would read the unlock rows of 0190_composite_lock_index_sync as locks. So undo them first, after “First, on both, while this release still runs” above.

  1. Stop what writes: note the replica counts first (kubectl -n <namespace> get deploy), then scale the Core worker and beat Deployments and the workflow engine’s Deployment to 0, for example kubectl -n <namespace> scale deploy/<release>-neops-core-worker deploy/<release>-neops-core-beat --replicas=0 (beat exists only with beat.enabled).
  2. Undo the migrations. Until step 3 replaces them, this release’s own pods fail most requests, so run step 3 right after.

    kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145
    kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py migrate core 0188_refreshtoken_auth_provider
    

    These are the last migrations of 2.2.0; for another release, ./manage.py showmigrations core cron in its image names them.

  3. helm -n <namespace> rollback <release> <revision> (helm -n <namespace> history <release> lists the revisions). Then check with kubectl -n <namespace> get deploy that the worker, beat and engine are back to the counts you noted; scale them back if not.

  4. Rebuild the earlier release’s indices and delete the ones only this release uses, with these three commands. Done when the rebuild exits 0 and a device search returns devices. After a rollback to 2.2.1-beta.2 to 2.2.1-beta.5, which already use those indices, run only the first two. After a rollback to 2.2.1-beta.7, which does not, run all three.

    kubectl -n <namespace> delete job <release>-neops-core-migrate --cascade=foreground --ignore-not-found
    kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py elastic_index --rebuild --force
    kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py shell -c "from elasticsearch_dsl.connections import connections; connections.get_connection().indices.delete(index=['network', 'device_group_edges_by_device_lookup', 'device_group_edges_by_group_lookup', 'client_group_edges_lookup', 'client_interface_edges_lookup'], ignore_unavailable=True)"
    

The next upgrade applies the migrations again and finds every index stale, so it rebuilds them all; the index delete frees the space of indices nothing reads any more.

By --atomic. A rollback by --atomic, or any helm rollback without step 2, leaves this release’s migrations applied, and the older image cannot undo migrations it does not know. Until step 2 has run, the older release cannot create entities. First stop the migrate Job with the first command of step 4, since it keeps rebuilding and retrying for the release you left, and stop the writers as in step 1. List the rows this release set aside while their status column still exists, since the older Core has no index_sync_status, with this query in the Core database (the Helm bullet under “Where it runs” says how to run one): SELECT 'device', id FROM core_device WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'interface', id FROM core_interface WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'group', id FROM core_devicegroup WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'client', id FROM core_client WHERE elastic_sync_status = 'rejected' UNION ALL SELECT 'execution', id FROM core_execution WHERE elastic_sync_status = 'rejected'; Then run step 2 from a copy of a Core pod that runs this release’s image, the one you noted (unnoted, it is quay.io/zebbra/neops-core:<the version you upgraded to>, or the image.tag you set, neops-core.image.tag under the umbrella chart). The copy keeps the pod’s environment and its command, which only starts the API, and drops its labels, so the Service sends it no requests:

POD=$(kubectl -n <namespace> get pod -l app.kubernetes.io/name=neops-core,app.kubernetes.io/instance=<release>,app.kubernetes.io/component=app -o name | head -1)
kubectl -n <namespace> debug "$POD" --copy-to=neops-core-migrate-back --set-image='neops-core=<this release's Core image>'
kubectl -n <namespace> wait --for=condition=Ready pod/neops-core-migrate-back --timeout=5m
kubectl -n <namespace> exec neops-core-migrate-back -- /docker-entrypoint.sh ./manage.py migrate cron 0015_add_execution_log_retention_periodic_task_20260201_0145
kubectl -n <namespace> exec neops-core-migrate-back -- /docker-entrypoint.sh ./manage.py migrate core 0188_refreshtoken_auth_provider
kubectl -n <namespace> delete pod neops-core-migrate-back

Then run the rest of step 4, and bring the writers back: kubectl -n <namespace> scale deploy/<release>-neops-core-worker deploy/<release>-neops-core-beat deploy/<engine deployment> --replicas=<the counts before step 1> (one command per count, if they differ). Or run helm -n <namespace> rollback <release> <the revision --atomic restored>, then check the counts as in step 3. A rollback after which neither step 2 nor the index delete ran cannot be detected: the next upgrade trusts the indices the older Core wrote under, and can show a restricted user a device whose access was revoked meanwhile.

Centralized logging

https://github.com/zebbra/neops-core/pull/2409 brings Core’s half of the centralized logging design; the workflow engine, worker SDK and neops-docker-compose follow it with the same settings and record keys.

  • Stores. NEOPS_LOG_BACKEND picks the writer: db (DbLogger), victorialogs (the new VictoriaLogsLogger, https://github.com/zebbra/neops-core/pull/2355) or loki. NEOPS_LOG_CLASS names any class explicitly, such as neops.core.log.loggers.optimized_db.OptimizedDbLogger. Loki is pushed through Core’s own emitter with structured metadata; python-logging-loki is gone. VictoriaLogs splits a message above about 200 KB into [part i/n] lines, since VictoriaLogs silently drops lines above 256 KiB.
  • Record keys. Lines written to VictoriaLogs and Loki carry the canonical record keys shared with the workflow engine, including the workflow run (workflow_execution_id) and the keys for function block identifiers that the engine and worker fill in. Core itself writes the workflow run on the lines of its lock mutations: the lock reference is bound as workflow_execution_id, and the unlock audit lines keep it. LogEntry stores it in a new column (migration 0192_logentry_workflow_execution_id).
  • Log context. Every task log line, and every console line of the API, the Celery worker and beat, carries the ambient log context and a per-process seq.
  • Reading. Query.logging needs execution:read or the new workflow-execution-log:read, which the operator and admin profiles grant. Lines of a workflow run are returned only with workflow-execution-log:read. Its query argument is now an escaped, case-insensitive substring filter on every store instead of a raw store query, newest and oldest are no longer swapped, and a store that cannot be queried answers queryable=false.
  • Lines survive rollbacks. OptimizedDbLogger buffers writes and flushes them from a background thread in autocommit, so lines logged inside a transaction that rolls back are kept. A line whose linked row is not committed yet is retried for 30 seconds and then written without that link, never dropped. DbLogger now also keeps lines that carry no execution, such as the attribution line of each change applied under a lock (https://github.com/zebbra/neops-core/pull/2397), and retention purges them.
  • Unlock audit lines are written after the unlock commits, so they are no longer atomic with it.
  • Providers. generic-rest-facts v2 and v3 and generic-netbox-jinja-facts have a per-task Log Level (DEBUG, INFO, WARNING; https://github.com/zebbra/neops-core/pull/2282). INFO logs one line per request, DEBUG adds bodies and facts, capped at 200 KB, and failed requests are logged at WARNING with their body. Saved tasks keep working: disable_log maps to WARNING and log_long_returns to DEBUG. Lines below a task’s level go to the container log instead of disappearing (DEBUG lines are dropped), and NEOPS_TASK_LOG_LEVEL sets the default (https://github.com/zebbra/neops-core/pull/2286).
  • Templates. Provider templates can write to the task log with neops.log(message, level) and neops.log_debug/info/warning/error (https://github.com/zebbra/neops-core/pull/2283), and stop on purpose with neops.fail(reason) or skip an element with neops.skip(reason) (https://github.com/zebbra/neops-core/pull/2285).
  • Metrics. The enterprise neops_metrics plugin (https://github.com/zebbra/neops-core/pull/2213) exports task, log size and retention metrics, and optionally database and table sizes; /metrics needs a credential, see “Metrics endpoint needs a credential”.
  • Performance. Execution results and log entries are written in batches, retention raw-deletes the execution cascade, and Nornir hosts load without a platform query per device (https://github.com/zebbra/neops-core/pull/2367).

Other changes

Locking and workflows

Authentication and API

Deployment

Inventory and providers

Dependencies

The search documents link related rows by id instead of embedding them, so an aggregation such as groups.name on devices is now computed from the related indices. Every field develop’s documents mapped aggregates as it did on develop: a row counts once per distinct value, values keep their type, and buckets are ordered by count, then value. A text field such as groups.name still buckets by word (zurich, hq); groups.name.raw buckets by name.

Three results differ from develop, on purpose:

  • groups.location.<field> on devices counts each device once per location it is in. Develop returned no buckets: its device document never carried a group’s location.
  • A user with group-restricted roles sees the clients those groups grant in client aggregations. On develop the restriction matched no client at all.
  • A related entity’s field that develop’s embedded copy left out (for example device.serial on interfaces) now aggregates instead of returning no buckets.

Two more follow from where a related entity’s dynamic facts (facts.*, checks.*) are read. They are now read from that entity’s own index and keyed by its mapping; develop read the copy embedded in the searched entity’s document, keyed by the searched index’s mapping, which took each fact’s type from the first linked copy indexed there.

  • Facts named like an IP address. The device, interface and group indices map a fact named ip, ipv4, ipv6, ip4, ip6 or ipaddress, in any letter case (IPv4), as an IP address; the client index keeps it as a plain string. So the paths that cross between clients and the other entities key it differently from develop:
  • clients by interface.facts.ipv4, interface.device.facts.ipv4, interface.device.groups.facts.ipv4, interface.device.groups.location.facts.ipv4 or location.facts.ipv4: one address written two ways (10.0.0.1 and ::ffff:10.0.0.1, 2001:DB8::A and 2001:db8::a) is now one bucket in its normal form, and buckets with equal counts are ordered by address. Develop kept them apart, as written, and ordered them as text.
  • interfaces by clients.facts.ipv4, devices by interfaces.clients.facts.ipv4, groups by devices.interfaces.clients.facts.ipv4: the other way round. Develop merged the spellings and ordered by address; now they are kept apart, as written, and ordered as text (10.0.0.10 before 10.0.0.9).

With a size limit, the top buckets can differ accordingly. Only paths where exactly one of the searched entity and the entity holding the fact is a client differ; interfaces by clients.location.facts.ipv4, for example, is unchanged. - Facts whose values differ in type between rows (speed: "auto" on one interface, speed: 1000 on another; and devices and interfaces share one mapping, so also speed: "fast" on a device). The related index types such a fact by the first document that carried it, which can be a row the searched entity never reaches (an interface without clients, a group without devices) or, for devices and interfaces, the other entity. Values can then be keyed as strings ("1000", ordered as text) where develop keyed numbers or dates, or the other way round.

Develop also refused the whole searched document when a linked row’s fact did not fit the type the searched index had given that fact: with speed: 1000 on one device’s interface and speed: "auto" on another’s, develop’s devices index refused the second device, which then dropped out of every device search and aggregation, and its clients and groups indices could refuse that interface’s clients and the device’s groups the same way. Such rows now count; only the interface itself can be refused, as below, as develop’s interfaces index refused it.

Separately, and not caused by aggregation: devices and interfaces share one mapping in the network index (https://github.com/zebbra/neops-core/pull/2379), so Elasticsearch refuses, as a whole, a device or interface whose fact or check does not fit the type the other entity gave it, in either direction (an interface with mtu: "jumbo" after a device with mtu: 1500). Develop kept device and interface facts under separate mappings. What follows from a refusal:

  • Searches and aggregations leave out a new row, and show an existing row with its old values: a rename or a group change does not reach search. So do the paths that read that row’s document from another entity: interfaces by device.<field> (not device.id); clients by interface.<field> (not interface.id) and by interface.device.<field>; devices by interfaces.<field>; groups by devices.<field> and by devices.interfaces.<field>. Paths that go on through the database (device.groups.*, interface.device.groups.*, interfaces.clients.*, devices.interfaces.clients.*) and interfaces by device.id and clients by interface.id still count the rows behind it.
  • A mutation that saves such a row (deviceUpsert, deviceAddToGroups) succeeds once the database commits: its response carries extensions.indexing = "pending", and the row stays in error until the periodic index sync task, or manage.py resync_unsynced_rows, sets it aside as rejected. Only a script that calls save() outside a transaction still fails with “1 document(s) failed to index.”, although the change is saved.
  • A fact a workflow writes through an unlock’s db_updates: the unlock succeeds, but every lock that searches the index is then refused with INDEX_SYNC_PENDING until the document has been rejected NEOPS_UNLOCK_INDEX_SYNC_MAX_ATTEMPTS times (default 3, at least NEOPS_UNLOCK_INDEX_SYNC_GRACE_SECONDS, 60 s, apart). From then on every lock that resolves that device or interface, by id or by query, is refused until its document is accepted: fix the fact and run resync_unlock_index --retry-quarantined, or give it up with resync_unlock_index --clear <reference> (the command lists the references).
  • Once set aside, the row is logged (“Elasticsearch rejected the document of Device “). elastic_index --rebuild sets it aside too, as the upgrade notes above describe.

Sorting by a related entity’s field works again as on develop: interfaces by device.id, device.hostname and device.ip; clients by interface.id, interface.name, interface.device.id, interface.device.hostname, interface.device.ip and location.id, location.name, location.address, location.city, location.country, location.lat, location.lon.

While the indices are being rebuilt, aggregating or sorting by a related entity’s field fails with “The search indices are being rebuilt” instead of combining half-filled indices, and after a rebuild that failed or was killed part way, until a rebuild succeeds, with a message that names it and the rebuild command; a field of the entity itself still answers. When a search index is missing, an aggregation now fails, as lists already did, with “Search is not available right now: an index it needs is missing or being rebuilt” and the command that rebuilds it, instead of showing 0. Without https://github.com/zebbra/neops-docker-compose/pull/27, ./neops up does not repair a missing network, clients or groups index: it creates it empty, and until the indices are rebuilt, searches miss those rows without saying so, and a filter on one of their facts through a related entity (groups.facts.site:zh on devices) fails with “Unknown column [facts.site]”. With it, up’s --rebuild-if-stale rebuilds such an index. A missing lookup index ./neops up recreates and refills either way. Any other error Elasticsearch answers with reports its reason, or its type, with the index when Elasticsearch names one; a connection error or timeout still shows the client’s own message.

An archived row that a forward link still reaches (an imported client’s archived interface) has no document, so its values are read from the database and keyed as Elasticsearch would have keyed them under the field’s mapping. A value Elasticsearch would have refused ("1G" in a numeric fact) is left out and logged as “Left out archived …” on every request that reads it. A fact that only archived rows carry has no mapping: it is keyed as dynamic mapping would have mapped its first value, in row order; develop took the first document indexed, so a fact with values of different types can key differently.

Develop answered each of these in one request. Now they read the related rows for every request, so they cost more as those tables grow, and faster than linearly at a million rows. Measured on one Elasticsearch node with 5,000 devices, 100,000 interfaces and 50,000 clients; the first run is what a dashboard pays after the index changed, a repeated one is answered partly from Elasticsearch’s request cache:

Aggregation or sort First run Repeated
a field of the entity itself, groups.<field> on devices, device.<field> on interfaces, location.<field> on clients under 50 ms under 50 ms
interfaces.<field> on devices 35 ms (name), 0.4 s (id) 4 ms, 0.2 s
clients.<field> on interfaces 1.1 s 0.4 s
interfaces.clients.<field> on devices, devices.interfaces.<field> on groups 0.6 s 0.2 to 0.5 s
interface.<field> and interface.device.<field> on clients 0.4 to 0.8 s 0.4 s
clients sorted by interface.name or interface.device.hostname 0.5 s 0.5 to 0.7 s

backend/neops_modules/opencore/neops_core/neops/core/documents/ELASTICSEARCH.md, section 7, has the details.

How index sync behaves now

Writes follow the commit

A change writes to the database inside its transaction and only marks the row pending there. The Elasticsearch write runs after the commit and reads the committed row. A rollback, or a process that dies before the commit, leaves search unchanged.

Restricted users and pending rows

When a change alters who may see a row (a device’s group membership, a client’s groups, an interface moved to another device), the row is flagged until its index write lands. While it is flagged, users with group-restricted roles do not see it through search. Admins and unrestricted users always see it.

Other changes, such as discovery updating facts, do not hide anything.

If more than 10,000 rows of one type wait for such a write (NEOPS_PERMISSION_PENDING_EXCLUDE_CAP), a restricted search fails with “Search is catching up after an indexing failure” instead of returning an incomplete list. The message names what brings the number down: the next workflow lock request, the periodic task where Celery beat runs, or manage.py resync_unsynced_rows. Rows hidden from restricted users that Elasticsearch rejected count too, and wait until their data is corrected and the row saved; manage.py index_sync_status --list rejected --limit 0 names them.

Repair after an Elasticsearch outage

A row whose index write failed stays error (or pending if the process died before writing). A device group whose documents a membership change has not rewritten yet is recorded the same way. Core re-indexes such rows and groups once NEOPS_UNLOCK_INDEX_SYNC_GRACE_SECONDS (default 60) have passed since the row’s status last changed (elastic_status_changed_at), so a change’s own write gets there first and a row that just failed is not retried at once. The longest waiting go first. It runs:

  • on every lock request from the workflow engine, a few rows at a time (NEOPS_INDEX_RESYNC_INLINE_BUDGET, default 20);
  • from the periodic unlock index sync task, when Celery beat runs (Helm: on by default; neops-docker-compose: the cms-tasks profile);
  • on demand with manage.py resync_unsynced_rows [--budget N].

A run stops at the first sign that Elasticsearch or the database is unavailable. After that, lock requests wait NEOPS_INDEX_RESYNC_INLINE_BACKOFF_SECONDS (default 60) before trying again.

Runs that overlap share the work. A run leases the rows it takes, 20 at a time, for 30 seconds: other runs skip them meanwhile, and rows leased by a run that died are taken again once the lease ends. The periodic task runs one at a time; a run that finds another still running exits at once, and each run stops after 45 seconds. It holds a PostgreSQL advisory lock in an open transaction while it runs, so a server whose idle_in_transaction_session_timeout is below 45 seconds ends that session mid-run and lets the next run start alongside it; the row leases still keep the two from writing the same rows. A lock request that resolves its entities through search during an index rebuild is refused before it repairs anything.

Repairing a row writes its own documents and lookup rows, then the counts of the rows it names itself (an interface’s device, a client’s interface, a device’s groups), once per run. A group’s members are not rewritten by repairing the group: membership changes, archives and restores write them. Selecting the rows to repair reads a partial index (migration 0201_resync_selection_indexes), so a lock request with nothing to repair no longer scans the interface table.

A document Elasticsearch rejects as such (a mapping or parse error) is marked rejected and skipped until the row changes again. The reason is logged when Elasticsearch rejects the document and not again while the row waits, so if a manual resync_unsynced_rows run set the row aside, the sweep’s line is in that run’s output; the write that failed first logged the same reason as ES sync failed for <row> where the change was made.

A row that fails for a reason of its own, such as data that cannot be serialized, stays error until it changes and is logged with its traceback. The run goes on with the next row, and the row is tried again the same wait after that try (elastic_sync_retry_at). Its elastic_status_changed_at keeps the time it first failed, so a row that keeps failing shows its real age.

Runbook: search disagrees with the database

The commands are shown for neops-docker-compose. With Helm, run kubectl -n <namespace> exec deploy/<release>-neops-core -- /docker-entrypoint.sh ./manage.py <command> instead; from a source checkout, uv run python manage.py <command> in backend/.

  1. Re-index what Core knows is behind:
./neops compose -- exec -T cms python manage.py resync_unsynced_rows

It prints how many rows it re-indexed, set aside, left for the next sweep and failed on. It exits 1 if Elasticsearch or the database is unavailable, and 3 if rows failed for a reason of their own (the log names them; they are retried later). Exit 3 here is not the rebuild’s 3: it means rows failed on their own data and will be retried. It takes every unsynced row at once, without the wait, except the rejected ones.

  1. If search is still wrong, refill the indices from the database without deleting them:
./neops compose -- exec -T cms python manage.py elastic_index --populate --no-sync-lookups
./neops compose -- exec -T cms python manage.py sync_lookup_indices --populate-only
  1. Only if an index has the wrong mapping (for example after Elasticsearch created it on its own), rebuild it. An index Elasticsearch created has no build record, so --rebuild-if-stale rebuilds it, with the indices it shares or mirrors, and leaves the others alone. This takes search down while it runs:
./neops compose -- exec -T cms python manage.py elastic_index --rebuild-if-stale