Skip to content

Neops Core 2.4.0

Released 2026-11-07

These notes cover network zones, which the workflow engine uses to send each job to a worker that can reach its device, and core log lines that now survive a log store outage.

At a glance

Area What changed Why it matters
Network zones A new NetworkZone (name and description), and an optional zone on every device. A device with no zone stored is in the zone default. Two isolated networks can use the same addresses; the zone keeps their devices apart.
Managing zones Zones are created, described and deleted in Django admin, on the web client’s Network zones page (web client 6.1.0) and with the GraphQL mutations networkZoneUpsert and networkZoneDelete. Create a zone before you start its workers.
Permissions The mutations need write on the new permission element network-zones, which no role has by default; bootstrap_admin_role gives it to the admin role, also when other users hold that role. Reading zones needs no grant. Grant network-zones write to the other roles that manage zones.
Moving devices Neither admin nor deviceUpsert can set a device’s zone; admin shows it read-only, default included. A device changes zone only through a workflow run. Devices created in admin, by deviceUpsert or by the importer are in default.
Writes from workflow runs mapByFieldOrdered is a list of identifier groups, such as [["ip", "network_zone"]]. An interface’s device reference resolves inside the zone it names. Discovery in one zone no longer binds a device of another zone that shares its address.
Export The neops_data export writes a device’s zone name as network_zone, and nothing for a device in default. The import does not read it yet. Re-importing an export puts every device in default.
Search The device search document carries network_zone with the zone’s id and name, and network_zone.name: default for a device with no zone. A selector can pick one zone’s devices exactly, for example ip:"10.0.0.1" AND network_zone.name:zone-b.
Locks The engine’s lock operation returns each locked device’s zone, and each locked interface’s device with that device’s zone. The engine knows every job’s zone when it locks.
Refused writes unlockResources answers a code: WRITE_REFUSED when a run’s writes can never be applied as sent, UNEXPECTED for any other failure, null on success. The workflow engine can end such a run instead of retrying it.

Upgrade notes

The upgrade applies migration 0206_network_zone. It creates the zone table and adds an empty network_zone column to devices, so every device starts in default. No device row is rewritten.

The network search index gains a field, so its schema hash changes and manage.py elastic_index --rebuild-if-stale rebuilds it on the upgrade with no further step. While the rebuild runs, and after a failed one until a later one finishes, core refuses every lock that searches the index; the workflow engine retries those locks, so a run that selects devices waits. Until the rebuild has run, devices with no zone are not indexed under default, so a selector on network_zone.name misses them. Run discovery, or any workflow that selects devices by zone, after the rebuild.

Going back below 2.4.0

Going back to an earlier release undoes migration 0206_network_zone, which deletes every zone and every device’s zone without a warning. Note which devices are in which zone before you go back: for each zone, query devices(networkZone_Name: "<zone>") { totalCount results { id hostname ip } }, and page with results(page: 2) and on past 200 devices. Going forward again, every device starts in default: recreate the zones, then move the devices back with fb.base.neops.io/device_update_device:0.2.0 runs, since a device changes zone only through a workflow run.

A role for the first superuser: bootstrap_admin_role

The new command manage.py bootstrap_admin_role --user <username> --role <role> gives that user the role, creating it if needed, with:

  • write on the scope Global (the first of NEOPS_SCOPES), whose own fields, such as its visibility flags, it leaves alone;
  • write on network-zones, so the user can create zones right after a fresh install.

Run manage.py grant_workflow_permissions --role <role> --profile admin --yes after it for the workflow grants. Both commands re-apply their grants on every run, so a grant removed from that role by hand comes back on the next one; give other users roles of their own. Without the neops_permissions_simple plugin it seeds nothing, since every user may already do everything.

If other users already hold a role of this name, the seed widens it only by the write on network-zones, which this release introduces, adds the user to it and exits 3, so a script that runs grant_workflow_permissions after it can skip that, as the Helm hook and cms-init do. It changes nothing else on a shared role: not its default permission, its scopes or its other grants. Whoever holds the role gets that write; with OIDC, the identity provider decides who that is. Zones are global, not per tenant, so a per-tenant IdP role named like superuser.role (NEOPS_ADMIN_ROLE) gives every tenant’s admins write on all zones. The command re-applies the grant on every install and upgrade: to keep it off a shared role, point superuser.role (NEOPS_ADMIN_ROLE) at a role only the superuser holds, then remove the grant from the shared role by hand.

With superuser.enabled, the Helm chart’s migrate hook now runs both for the superuser and the role superuser.role (default admin), on every install and upgrade. Before, the chart’s superuser could log in and then read and write nothing until someone granted a role by hand. neops-docker-compose’s cms-init runs the same two commands.

Zone names

A zone name is lowercase letters, digits and hyphens, at most 63 characters, and starts and ends with a letter or digit: zone-a, stadium-3. Names are unique, and default is reserved: no zone can take it, since a device with no zone is already in it.

A name is fixed once the zone exists. Admin shows it read-only, and networkZoneUpsert with an id refuses a different name. To fix a misspelt zone, create the right one, move its devices with a workflow run, then delete the old one. The description stays editable.

GraphQL

query {
  networkZones(name_Istartswith: "zone") {
    totalCount
    results { id name description }
  }
  devices(networkZone_Name: "default") {
    results { hostname networkZone { name } }
  }
}
  • networkZones lists the zones stored in the CMS. It never returns default, which has no row.
  • The generated clients’ networkZones operation takes page and pageSize, 1 and 200 by default, so a client can read more than 200 zones.
  • DeviceType.networkZone is null for a device in default.
  • The devices filter takes networkZone (an id) and networkZone_Name. networkZone_Name: "default" matches the devices with no zone.
  • networkZoneUpsert(name, description) creates a zone. It refuses a taken name, default, and a name off the pattern above.
  • networkZoneUpsert(id, description) changes the description.
  • networkZoneDelete(id) deletes a zone; see the next section.

Deleting a zone

A zone delete, in admin or with networkZoneDelete, is refused while a device that is not archived uses the zone, with the same message in both: Network zone zone-a is used by 2 device(s). Move or delete them first. Move those devices to another zone, or delete them, first.

Deleting a device only archives it. When a zone is deleted, its archived devices lose their zone in the same transaction, so an archived device you restore afterwards comes back in default.

Core cannot see which workers serve a zone or which runs still name it. To retire a site:

  1. While its workers still run, move its devices to another zone, or delete them.
  2. Stop the site’s workers.
  3. Delete the zone, in Django admin or with networkZoneDelete. While a live device still uses it, the delete is refused with the number of devices that use it.

If the site is already unreachable, its devices can still be moved. Run fb.base.neops.io/device_update_device:0.2.0 with network_zone set to the target zone, in a step whose zone is {name: <target>, override: true}, so a worker that still runs picks the job up. The block doesn’t connect to the device, and override only decides where the job runs; the network_zone parameter is what moves the device.

Writes from workflow runs

The workflow engine applies a run’s creates, patches and deletes through unlockResources, in order and in one transaction, so a move earlier in the run counts for the writes after it.

A refused write: WRITE_REFUSED

When core refuses a run’s writes, unlockResources answers success: false, lockExists: true and a message, as before, and now also a code. Nothing of the run is written and the lock stays held.

  • WRITE_REFUSED: the same writes would be refused again on every attempt. Examples:
  • a value a field can’t take, such as a string where a relation needs an id or {"id": ...}, or a reference that matches no entity or more than one;
  • a field the entity doesn’t have: an unknown PATCH path such as nosuchfield or hostname.x, or an unknown key in a CREATE’s data;
  • a create an identifier group refuses;
  • a value another entity already holds where it must be unique, such as a group’s name or a device’s external_reference, archived entities included;
  • a patch or delete of an entity outside the lock;
  • a dbUpdates payload that doesn’t validate, such as a PATCH without path. Before, that came back as a GraphQL error instead of an answer.
  • UNEXPECTED: anything else went wrong, such as a lost database connection, or two runs writing the same unique value at the same moment. Trying again can succeed: in that race the retry finds the other run’s entity (with skipIfExists), or is refused with WRITE_REFUSED.
  • On success, and when the lock doesn’t exist (lockExists: false), code is null.

The message names the entity, for example DEVICE 42 is not locked by this lock, Group with name 'core' already exists, Invalid value for device (id 42) at nosuchfield: invalid path nosuchfield or, for a create, Invalid value for device (new): ... (while setting 'platform'). Core logs every refusal as a warning:

Unlock 'wf-exec-123' for owner 7 refused (WRITE_REFUSED): Group with name 'core' already exists

What happens to the run depends on the workflow engine:

  • An engine that acts on WRITE_REFUSED ends the run FAILED_SAFE with that message. Such an engine requires this core version.
  • An older engine ignores code and keeps retrying a refused unlock with the locks held, as before.

Upgrade core first. An engine built on the client that selects code fails every unlock against an older core with Cannot query field 'code' on type 'UnlockResources'. The engine will raise its minimum core version to guard against that.

code is a LockErrorCode, the enum lock refusals already use. The operation in neops-graphql/graphql/unlock_resources.graphql selects it.

Identifier groups

A create’s mapByFieldOrdered is a list of groups. The fields of a group must all match. Groups are tried in order, a group is skipped when the data lacks one of its fields, and the first group that finds an entity decides: one match is reused when skipIfExists is set and refused otherwise, two or more are refused.

{
  "operation": "CREATE",
  "entity": "DEVICE",
  "id": -1,
  "data": {"hostname": "r1", "ip": "10.0.0.1", "network_zone": {"name": "zone-b"}},
  "mapByFieldOrdered": [["ip", "network_zone"], ["hostname", "network_zone"]],
  "skipIfExists": true
}
  • This create looks 10.0.0.1 up only among the devices of zone-b. {"name": "default"} matches the devices with no zone.
  • A group without network_zone, such as [["ip"]], matches across every zone, as before.
  • Core still reads a bare field as a group of one, so ["ip"] means [["ip"]] to unlockResources. The workflow engine does not: it refuses a job’s result whose identifierFields holds a bare field (identifierFields must be a list of field groups), and the run fails with invalid_result. The worker SDK 0.5.0 turns a bare field into a group of one before it sends the result; a hand-built result writes [["ip"]].
  • Core checks the whole list before any lookup and refuses the write if a group is empty, names a field the entity can’t be identified by, or holds network_zone alone. A device is identified by ip, hostname and network_zone; an interface and a group by name.
  • These fields are now set per entity, and mac_address is no longer one of them. Before, one list served every entity and included mac_address; a hand-built result that names mac_address in mapByFieldOrdered is now refused.

A device’s zone on a create or a move

  • A device create carries network_zone: {"name": "zone-b"}. A create without network_zone, or with {"name": "default"}, stores no zone: the device is in default. Neither core nor the workflow engine refuses it, from a job in any zone: the engine passes the create as sent and does not fill in the job’s zone. A device that a job in zone-b creates without network_zone is in default: a later job on it runs on a worker of default, unless its step overrides the zone, and that worker gets its credentials. The worker SDK 0.5.0 requires network_zone on a device create; a hand-built result must name it, normally the job’s zone.
  • A patch with path network_zone and value {"name": "zone-a"} moves a device.
  • A zone is named as {"name": ...} or {"id": ...}. A null id counts as absent, so {"id": null, "name": "zone-b"} names zone-b. A zone that names nothing, such as {"id": null}, or a zone that doesn’t exist is refused with a message naming device.network_zone.

Interface device references

An interface names its device as {"id": ...}, or as {"hostname": ..., "ip": ..., "network_zone": {"name": ...}} with a hostname, an ip or both. Core resolves it on every field it carries, inside the zone it names, both in the interface create’s existence check and when it sets the interface’s device. An id wins over the other fields and may be the temporary id of a device created earlier in the run. A reference that matches no device, or more than one, is refused with a message naming it. A reference without network_zone is matched across every zone.

Search by zone

Each device in the network index carries network_zone: {"id": ..., "name": ...}. A device with no zone is indexed with the name default and no id, so a selector names default as it names any zone.

ip:"10.0.0.1" AND network_zone.name:zone-b
network_zone.name:default
device.network_zone.name:zone-b
  • The name is a keyword, so network_zone.name:acme matches the zone acme and not acme-dr. A wildcard such as network_zone.name:acme* matches both.
  • The last line is an interface selector: it picks the interfaces of zone-b’s devices.
  • A selector without network_zone.name matches every zone, so two devices that share an address in different zones both match ip:"10.0.0.1".
  • Device lists sort by zone with the ordering key networkZone.name (-networkZone.name descending). A device with no zone sorts as default, by that name among the others. The mapping is unchanged, so this needs no index rebuild beyond the one above.

What a lock returns

getAndLockResources in neops-graphql/graphql/get_and_lock.graphql, the operation the workflow engine locks with, now selects:

  • networkZone { id name } on each device;
  • device { id networkZone { name } } on each interface.

networkZone is null for a device in default.

Known limits

The legacy 1.0 discovery and the neops_data import are not zone-aware. The devices they create are in default, and they match existing devices by hostname or IP across every zone. Once zones other than default hold devices, running them is unsupported: with an address shared between zones they can update another zone’s device, or fail on two matches. Making them zone-aware is a separate task.

Log lines survive a log store outage

With NEOPS_LOG_BACKEND=victorialogs or loki, core used to try a batch three times within about a second and then drop it. A store outage lost every core line of that window, including the lock <ref>: applied <OP> on <entity> <id> -- actor: ... line that records who applied a change.

Core now keeps those lines in a new database table and replays them, oldest first, once the store takes a push again. This matches the workflow engine:

  • Not while your request waits. Pushes run on a background thread, so a slow or missing store no longer delays a request or a Celery task. A query for logs waits at most 5 s for lines that are still on their way.
  • A rollback keeps the line. The outbox is written on its own connection, outside the caller’s transaction.
  • No Celery needed. Every core process that serves requests replays, so this also works without the cms-tasks profile.
  • Bounded. The outbox holds at most NEOPS_LOG_OUTBOX_MAX_ROWS lines (500000) and NEOPS_LOG_OUTBOX_MAX_BYTES bytes (1 GiB). Past either cap the oldest lines are dropped and a warning is logged.
  • A bad line cannot block the others. A line the store refuses on its own is dropped with a warning. A line that keeps failing while the store takes other lines is dropped after NEOPS_LOG_OUTBOX_MAX_ATTEMPTS tries (10).

A line can reach the store twice if the store took a push but its answer never arrived, or if core died before it deleted the replayed rows. The engine accepts the same. A line moved to Loki’s 50-minute bound (see below) takes a new timestamp on each push, so Loki cannot recognise a retry after a timeout as the same line and can store it twice.

Loki keeps a line only within a window. It refuses a line more than an hour behind the newest line of its stream, and one older than a week. After a longer outage core writes such a line at 50 minutes ago, and keeps the time it was logged in the event_time field. The engine does the same. So after an outage of more than 50 minutes, a replayed line sorts by when it was delivered, not when it happened, and event_time has the real time. If Loki still finds lines too old, for example with a shorter max_chunk_age or a clock far off, core does not retry that batch: it counts the batch under too_old, as “at most this many”, because Loki stores the lines of a batch it did not refuse. VictoriaLogs has no such window, but it silently ignores a line older than its own retention.

Run manage.py log_store_status to see how many lines wait, how old the oldest one is and how many were dropped per reason. Run it twice to see whether the backlog grows or drains. --json prints the report on one line. It does not say whether a process currently considers the store down: that is logged once, when it changes. After a change of NEOPS_LOG_BACKEND, lines waiting for the old backend stay in the table under their backend’s name until the caps remove them, and no process replays them.

Sizing. These figures are estimates, not measurements. The caps count the JSON text of the lines, not what Postgres uses for them, so allow roughly 1.5 to 2 times NEOPS_LOG_OUTBOX_MAX_BYTES on the Postgres volume: 1.5 to 2 GiB at the default. Deleted rows keep their space until autovacuum reclaims it. To read the real size during an outage, run select pg_size_pretty(pg_total_relation_size('core_logoutboxentry'));. To size the cap, multiply your core lines per hour by the bytes of a line and by the outage you want to survive. A line is probably a few hundred bytes; during an outage, log_store_status gives your own figure as bytes divided by rows. Replay runs at up to about 2500 lines per second per process.

New settings, all optional: NEOPS_LOG_OUTBOX_MAX_ROWS, NEOPS_LOG_OUTBOX_MAX_BYTES, NEOPS_LOG_OUTBOX_MAX_ATTEMPTS, NEOPS_LOG_BATCH_MAX_BYTES (1 MiB) and NEOPS_LOG_QUEUE_MAX_RECORDS (10000). They have the same names and defaults as the engine’s. neops-docker-compose and the Helm charts do not pass them through yet, so changing one there needs an override.

Upgrade: migration 0205_log_outbox adds two tables and needs no downtime.

Going back: the procedure in the 2.3.0 notes runs migrate core 0188_refreshtoken_auth_provider with the new code. That removes both tables, with every line still waiting in them and the drop counts, and nothing warns about it. An older core does not replay these lines either. Before that step, with the log store up and core serving requests, run manage.py log_store_status until rows is 0. If the store cannot come back in time, the waiting lines are lost: save them first with pg_dump --table=core_logoutboxentry if you need them. Do not go back without draining while lines of an applied change wait there, because those are the only record of who applied it.