Migrating an installation from 2.0 to 3.0 (preview)

Preview. Zequent 3.0 is not released yet; this runbook describes the preview builds and will change before the release. Until then, run it only against a copy of your installation.

This is the operator's step-by-step guide for moving an existing 2.0 installation — the customer Docker Compose setup or the Zequent Helm chart — to 3.0. What changes and why, which breaking changes affect integrations, and which parts must move together is explained in Upgrading the platform to 3.0; read it first. OpenBao and Keycloak themselves are described in Secrets and sign-in.

• Overview
• 1. Prerequisites
• 2. Back up
• 3. Set up Keycloak and OpenBao
• 4. Change the configuration
• 5. Move users and sealed secrets
• 6. Upgrade the services
• 7. What the database migrations do
• 8. Verify
• 9. Roll back
• 10. Adapters and client applications

Overview

The upgrade is a maintenance window: all core services and the console stop on 2.0 and start on 3.0 together. Plan for the database migration, which rewrites timestamp columns and can take minutes on a database with a long history.

1. Prerequisites

2. Back up

Stop nothing yet; take the backups right before the maintenance window and again after the 2.0 services are stopped (step 6.1), so the second one has no writes missing.

Postgres (with TimescaleDB)

The database zequent_db holds the platform's data and, in its own schema, the Integration Hub's.

# Docker Compose
docker compose exec -T postgres pg_dump -U postgres -Fc zequent_db > zequent_db-$(date +%F).dump

# Kubernetes: in the pod of your Postgres
kubectl -n <namespace> exec <postgres pod> -- pg_dump -U postgres -Fc zequent_db > zequent_db-$(date +%F).dump

pg_dump may warn about circular foreign keys in TimescaleDB's catalog; that is expected. A restore needs the same TimescaleDB version and runs between timescaledb_pre_restore() and timescaledb_post_restore() (see Roll back). Test the restore once into an empty database before the window.

Redis

Redis holds license leases, sessions, the simulator's fleet and, on installations that came from older releases, no-fly zones the console kept only in Redis (connector imports them into Postgres, 3.0 still reads them from there). Take a snapshot:

# Docker Compose (the container has the password in REDISCLI_AUTH)
docker compose exec redis redis-cli BGSAVE
docker compose exec redis redis-cli LASTSAVE      # repeat until the timestamp changes
docker compose cp redis:/data ./redis-$(date +%F)

On Kubernetes, use your Redis installation's snapshot procedure (BGSAVE, then copy the RDB and AOF files from its volume).

OpenBao and Keycloak

Nothing to back up: 2.0 has neither. From the first 3.0 start on, back them up as described in Backup and restore, and keep the OpenBao seal key apart from every backup.

3. Set up Keycloak and OpenBao

Both run next to the 2.0 services without affecting them, so this step can be done before the maintenance window.

Keycloak

Keycloak imports the realm zequent (clients zqnt-console and zqnt-admin-console) on its first start. Replace the temporary administrator admin with a permanent one afterwards. Details: How people sign in.

OpenBao

Both create the Transit keys service-token, credential, export-signing and platform-kek under zqnt-transit/, and write each KV secret once, when its path does not exist yet, from the values you already have:

A pepper is the key under which tokens, codes and stream keys are stored as hashes. If your 2.0 installation had LIVE_STREAM_KEYS_PEPPER set, keep exactly that value. Generate the three new peppers once (openssl rand -hex 32) and never change them: the first 3.0 start hashes every existing webhook token and claim code under them. A pepper you do not set is generated by openbao-init / the chart, which is fine for the new ones and wrong for an existing LIVE_STREAM_KEYS_PEPPER.

After the first start, OpenBao is the source of these values: change one with bao kv patch -mount=zqnt <path> <field>=… and restart the workloads that read it.

4. Change the configuration

Docker Compose (.env.customer and docker-compose.customer.yml)

Take the 3.0 docker-compose.customer.yml and .env.customer.example and carry your values over.

The 3.0 core services additionally need these settings (every core service unless noted):

Open before release: the preview docker-compose.customer.yml starts OpenBao and Keycloak but does not pass the settings above to the core services yet. Until it does, add them to the core services yourself, or run the upgrade on the Helm chart.

Helm (zqnt-platform-v2 values)

Default passwords are refused

A 3.0 service started outside development mode refuses to start when any of these holds a value from Zequent's local development setup or an unfilled <PLACEHOLDER> of .env.customer.example:

The error names the setting, never the value. An installation that still runs on a password copied from a developer setup must change it before the upgrade (for Postgres: ALTER USER … PASSWORD, then the setting).

5. Move users and sealed secrets

The 3.0 connector refuses to migrate the database while anything that 2.0 kept in its own database has not moved:

On a 2.0 database both guards apply: the seeded system administrator alone is a password user without a Keycloak account, and every licensed organization has an activation sealed with EXPORT_PLATFORM_KEK.

Open before release: there is no supported way yet to move a 2.0 installation's password users into Keycloak and its license activations into OpenBao Transit. The preview builds that did this on their first start no longer do, and the tool that replaces them is not available yet. On a plain 2.0 database the 3.0 connector therefore stops at this point with … password user(s) have no Keycloak account yet … or … license activation(s) … are still sealed with EXPORT_PLATFORM_KEK, and the database is left half-migrated (restore the backup, see Roll back). This section will describe the supported steps before 3.0 is released. Do not upgrade a production installation until then.

The error messages name a way past each guard that gives the data up instead of moving it. It is not a migration — it costs every password and every activation — and is only meant for test installations; talk to Zequent before using it anywhere else:

• A user that is to lose its password: UPDATE schema_zequent.users SET password_hash = NULL WHERE id = …. After the upgrade admin-console creates their Keycloak account without a password; an administrator sets one in Keycloak, or the person uses Forgot password? once Keycloak can send email.
• An organization's single sign-on to configure again: UPDATE schema_zequent.identity_providers SET client_secret = '' WHERE organization_id = …, then set it up again on Single Sign-On after the upgrade (Microsoft Entra ID for an organization).
• A license activation given up: delete its row from schema_zequent.license_activation_credentials, then activate the organization's license again in the console (Organizations → Activate license).

6. Upgrade the services

6.1 Stop 2.0

Stop the five core services, the console and the Integration Hub. Keep Postgres, Redis, OpenBao and Keycloak running. Take the second backup (step 2).

6.2 Start 3.0

Docker Compose

docker compose --env-file .env.customer -f docker-compose.customer.yml \
  --profile openbao --profile keycloak pull
docker compose --env-file .env.customer -f docker-compose.customer.yml \
  --profile openbao --profile keycloak up -d openbao-seal-key openbao openbao-init keycloak-db-init keycloak
docker compose --env-file .env.customer -f docker-compose.customer.yml \
  --profile openbao --profile keycloak up -d connector-service
docker compose logs -f connector-service       # wait for the migrations to finish
docker compose --env-file .env.customer -f docker-compose.customer.yml \
  --profile openbao --profile keycloak up -d

Helm

One helm upgrade with the 3.0 chart, the changed values and the 3.0 image tags. OpenBao, its configure Job, Keycloak and its realm Job come up with it; the core services restart a few times until OpenBao is configured. Watch the connector:

kubectl -n <namespace> logs -f deploy/connector-service

Order and what to watch

1. The connector runs the database migrations on its first start (step 7). Let it finish before the other services start. Do not start 2.0 services against the database from here on: after the first migration it is no longer a 2.0 database.
2. live-data, remote-control, mission-autonomy, admin-console and the console start on the same 3.0 version — never mix 2.0 and 3.0 core services.
3. The Integration Hub starts last. On its first 3.0 start it moves every credential 2.0 encrypted in its database into OpenBao, then drops the encrypted column. It needs VAULT_ENCRYPTION_KEY (your 2.0 value) for that start only; without it, and with encrypted credentials present, it stops with … credentials are still encrypted in the database: set VAULT_ENCRYPTION_KEY once … and changes nothing. After its log shows moved N credentials from the database to OpenBao, remove VAULT_ENCRYPTION_KEY.

Open before release: the 3.0 Helm chart has no value for VAULT_ENCRYPTION_KEY any more. An installation whose Integration Hub holds credentials from 2.0 has to give the key to the integration-hub-backend Deployment by hand for its first start (and take it away again) until the chart offers this.

7. What the database migrations do

The connector migrates the database from the last 2.0 version to 3.0 in one go. Notable steps:

Each migration runs in a transaction: one that refuses or fails changes nothing itself, but the migrations before it stay applied.

On its first start admin-console also converts the stored operator stream keys to the 3.0 form. Stream keys a 2.0 installation issued with LIVE_STREAM_KEYS_PEPPER set cannot be converted and are revoked (logged as Revoked N stream keys …): cut new keys for the devices that used them.

8. Verify

9. Roll back

The migrations drop columns and tables (stored webhook tokens, password hashes, the identity provider tables, the Integration Hub's encrypted credentials) and change column types. The only way back to 2.0 is restoring the backup; everything written on 3.0 since is lost.

1. Stop the 3.0 core services, the console and the Integration Hub.

2. Restore Postgres into an empty database with the same TimescaleDB version:

   # Docker Compose; on Kubernetes the same commands in the Postgres pod
   docker compose exec -T postgres psql -U postgres -c 'DROP DATABASE zequent_db' -c 'CREATE DATABASE zequent_db'
   docker compose exec -T postgres psql -U postgres -d zequent_db \
     -c 'CREATE EXTENSION IF NOT EXISTS timescaledb' -c 'SELECT timescaledb_pre_restore()'
   docker compose exec -T postgres pg_restore -U postgres -d zequent_db --no-owner < zequent_db-<date>.dump
   docker compose exec -T postgres psql -U postgres -d zequent_db -c 'SELECT timescaledb_post_restore()'
   

   CopyCopied!

3. Restore the Redis snapshot (stop Redis, put the saved /data back, start it).

4. Put back the 2.0 configuration — .env.customer or values, including EXPORT_PLATFORM_KEK and VAULT_ENCRYPTION_KEY — and the 2.0 image tags, and start 2.0.

OpenBao and Keycloak may keep running; 2.0 does not use them. Edge and client credentials issued on 3.0 are not valid on 2.0.

10. Adapters and client applications

Adapters and client applications do not have to move with the platform; see What keeps working.

Open before release: with OpenBao, the 3.0 platform signs its calls into adapters with the Transit key service-token. Adapters on a 2.0 Edge SDK verify those calls only against the static ZQNT_PLATFORM_PUBLIC_KEY and may refuse them. Check your 2.0 adapters against the upgraded platform during the rehearsal; adapters on a 3.0 Edge SDK fetch the key set from ZQNT_PLATFORM_KEYS_URL.

• Adapters: move them one at a time to the 3.0 Edge SDK — Java · Python · Go. Set ZQNT_PLATFORM_KEYS_URL to https://<console-api>/api/admin-console/auth/service-keys; keep ZQNT_EDGE_TOKEN and ZQNT_PLATFORM_PUBLIC_KEY.
• Client applications: after the platform, to the 3.0 Client SDKs — Upgrading to 3.0.