Choose and pin a release, back up the database, and rehearse the migration before upgrading. Schema migrations are forward-only, so some rollbacks require a database restore and lose writes made after the backup.
Pin your version
Pin a release tag or digest for production. latest can change between pulls; restarting with a pull policy that refreshes it can introduce an unplanned version and migration.
Sim publishes images to GHCR, tagged by release alongside latest:
ghcr.io/simstudioai/simstudio
ghcr.io/simstudioai/realtime
ghcr.io/simstudioai/migrationsKeep app, realtime, and migrations on the same release. They share a schema; mismatched versions can expect missing columns or incompatible behavior. An application-only rollback needs verified compatibility with the migrated schema.
Upgrade procedure
Before applying an upgrade, configure a direct, non-pooled MIGRATION_DATABASE_URL on the migrator if the app uses a pooler. Session advisory locks do not survive transaction pooling. Pass the URL through the migrator's database Secret on Helm or its explicit environment on Compose; app.env or .env alone does not reach it. See Migration environment for the exact setup.
Read the release notes
Check the releases page for new required environment variables and breaking changes. When the chart's minor version moves, also read its README.md upgrade notes — chart upgrades occasionally rename or remove values keys.
Take a backup
Snapshot the database immediately before upgrading. Because migrations are forward-only, this snapshot is your only rollback path for schema changes.
# Managed Postgres — take a manual snapshot
aws rds create-db-snapshot --db-instance-identifier sim-db \
--db-snapshot-identifier "sim-pre-upgrade-$(date +%Y%m%d)"
# Bundled Postgres — the Helm chart's database is named `sim` by default
# (Docker Compose uses `simstudio`; the cloud example values files override to `simstudio`)
kubectl exec -n simstudio statefulset/sim-postgresql -- \
pg_dump -U postgres -Fc sim > "pre-upgrade-$(date +%F).dump"Rehearse against real data
Migration surprises are usually data-shaped rather than schema-shaped, so a staging run against a copy of production data catches far more than a run against an empty database.
Verify
Run the verification checklist. At minimum: sign in, open a workflow, execute it, upload a file, and confirm the background jobs are still firing.
Rolling back
Application-only rollback is possible when no migrations ran, or when you have verified the previous app is compatible with the migrated schema. Additive migrations alone do not prove compatibility:
helm rollback sim -n simstudio# Production Compose — set SIM_VERSION in .env so the pin persists, replacing
# any existing line rather than appending a second one. Substitute the previous
# release tag from the releases page.
docker compose -f docker-compose.prod.yml up -dThe local and Ollama stacks build the app from source and ignore SIM_VERSION, so roll those back by checking out the previous tag and rebuilding with the file — and, for the Ollama stack, the profile — that started the install:
git checkout "<previous-release-tag>"
# Local stack
docker compose -f docker-compose.local.yml up -d --build
# Ollama stack — the profile you installed with
PROFILE=gpu # or cpu
docker compose -f docker-compose.ollama.yml --profile "$PROFILE" up -d --buildRollback after an incompatible schema change requires restoring the pre-upgrade database backup. This loses writes made since the snapshot. Stop all writers before restoring:
- Stop traffic and all processes that can write to the database, including the app, realtime service, scheduled jobs, and any background workers.
- Restore the pre-upgrade database snapshot.
- Redeploy the previous image tag on the app, realtime, and migrations services, then resume the matching background workers and scheduler.
- Verify.
Run the verification checklist before reopening traffic. Preserve the original encryption secrets with the restored database.
When a migration fails
The app pods will not become ready — this is by design.
kubectl logs -n simstudio deploy/sim-app -c migrations --tail=200# Or the file that started your install.
docker compose -f docker-compose.prod.yml logs migrationsCommon causes:
| Symptom | Cause |
|---|---|
permission denied to create extension "vector" | The database user lacks superuser rights. Create the pgvector extension manually as an admin, then re-run. |
| Connection refused / timeout | DATABASE_URL wrong, or the database is not reachable from the pod. Check network policy and credentials. |
Timed out ... waiting for the migration advisory lock | Another replica's migrator is wedged holding the lock. This replica waited 30 minutes and gave up. Find the runner that is actually stuck and read its logs. |
SQLSTATE 55P03 — lock timeout on a large table | A long-running query is holding a lock the DDL needs. The migrator sets lock_timeout = 5s on DDL rather than queueing every other query on that table behind it. Drain traffic and retry during a quiet window. |
| Constraint violation | Pre-existing data conflicts with a new constraint. Capture the error, restore the pre-upgrade backup, and open an issue with the exact message. |
Do not manually edit the migrations table to skip a failed migration — the schema and Sim's expectations will diverge in ways that surface much later.
Migration reference
Migrations are Drizzle SQL files plus registered script (data) migrations, applied by a dedicated image in one locked session. A failure in either stage fails the migration container.
- Kubernetes — an init container on the app Deployment. Every app pod waits for migrations to complete before it starts, so a failed migration blocks the rollout instead of starting an app against a mismatched schema.
- Docker Compose — a one-shot
migrationsservice withrestart: nothat runs before the app.
Migrations are forward-only. Take the backup described in the upgrade procedure before applying them; there are no down-migrations.
Migrations apply in sequence, so skipping several releases at once generally works mechanically. It is riskier — those combinations get less testing, and you take several releases' worth of behavior changes in one step. Upgrade in smaller increments where you can.
How concurrent replicas are handled
Every app replica runs the full migrator — there is no leader. They are serialized by a Postgres session advisory lock, taken with a bounded try-lock loop rather than a blocking wait:
- One runner takes the lock and applies the pending files. The rest retry every 5 seconds.
- The applied files are journaled, so the runners that follow find nothing to do and exit.
- A runner that cannot get the lock within 30 minutes fails with a message naming the deadline.
A stuck runner holds up later replicas until their 30-minute lock-acquisition deadline. Inspect the earliest migration container's logs to find the holder; a later timeout may come from a runner that was only waiting.
Migration environment
These control the migrator and are set on the migrations container, not the app.
| Variable | Purpose |
|---|---|
MIGRATION_DATABASE_URL | A direct, non-pooled database connection URL (DSN). Falls back to DATABASE_URL. Setting it also arms the backend-pid check that aborts the run if the session — and with it the advisory lock — is silently recycled mid-migration; with only DATABASE_URL set, that check is skipped |
Set MIGRATION_DATABASE_URL if anything sits between Sim and Postgres. Session advisory locks and session-level SETs do not survive PgBouncer transaction pooling — the lock silently does not hold, and consecutive statements can land on different backends. Point it at the database directly, bypassing the pooler.
Supply the migration URL separately from the app URL:
| Deployment | How to pass MIGRATION_DATABASE_URL |
|---|---|
| Helm | The init container reads only the database Secret. Include the key in a Secret supplied through postgresql.auth.existingSecret or externalDatabase.existingSecret; app.env does not reach it. The chart-managed database Secret carries only the password, and there is no dedicated values key for the migration URL |
| Compose | Add the variable to the migrations service's explicit environment: list. Setting it in .env alone does not pass it to that service |
| Separate migration run | Run the migration image yourself with a direct DSN before upgrading |
Behavior that is fixed in the migrator and not configurable, but that explains what you see in the logs:
| Behavior | Value |
|---|---|
| Advisory-lock acquire deadline | 30 minutes, retried every 5s |
lock_timeout on the migration session | 5 seconds for ordinary DDL — a statement that waits longer fails with SQLSTATE 55P03. An individual migration can set it to 0 for statements that must not be interrupted, such as CREATE INDEX CONCURRENTLY. statement_timeout is 0 on the same session by default, so a long migration is not cut off once it holds its locks — though a migration can narrow it for its own statements, as 0076_damp_vector.sql does with a 180-second local setting |
| SQL migration retry budget | 20 minutes, with exponential backoff and jitter, and only for a lock timeout (55P03). Script (data) migrations run once — a failure there exits immediately. They also run with lock_timeout = 0 and statement_timeout = 0, so one blocked on an app-held lock waits indefinitely: a stuck script migration looks like a hung migration container, not a 55P03 |
| Connection attempts at startup | 10, for transient failures — 53300 (too_many_connections), 53400, the 08xxx connection-exception class, and socket-level errors such as ECONNREFUSED, ECONNRESET, ETIMEDOUT, EHOSTUNREACH, and ENOTFOUND. A database that is simply down is retried with backoff, not failed immediately |
The lock_timeout deliberately trades a table-wide stall for a failed migration: without it, DDL waiting on an AccessExclusiveLock queues every other query on that table behind it for the whole wait.