Cutover Runbook

How Sondely's data moves from Cosmos (Mongo API, account cosmos-catalyzeup, databases impactpulse for dev and impactpulse_prod for prod) to Postgres in one cutover, how the move is proven before anyone depends on it, and how it is undone. The model it fills is on Data Model; the infrastructure it runs on is on Infrastructure and Deploy; the plan is on the overview.

The data moves big-bang: one write freeze, one migration, one switch of sondely.com to the new app. The code is proven before that day by the tests of the new app (Testing and Bugs), the rehearsals and the verification below.

State on 8 November 2026. The migration script and its verifier are built in CatalyzeUp/sondely-platform (scripts/migrate/). Rehearsal 1 ran on dev: migrate, then verify, against the real Cosmos dev database and the new dev Postgres, with counts, record hashes and recomputed scores all matching and nothing unresolved. The production data was then copied, as a rehearsal, from Cosmos impactpulse_prod into the new production Postgres (sondely_prod on psql-catalyzeup-sondely), and verification passes (Rehearsal 2). The Python app at sondely.com stays live and untouched, so that copy goes stale from the moment it was made; the migration is idempotent and is run again at the cutover. The new app is deployed: platform.sondely.com serves commit a003202 from main with database: ok, and platform-dev.sondely.com serves dev. The production reminder job is off on purpose (sondely_prod_reminders_enabled = false in catalyzeup-devops azure/sondely-platform.tf): the Python app still sends reminders to the same people, and both together would remind them twice. The cutover has not run; when it runs is Chip's call.

The new app runs in its own container apps (sondely-platform-dev and sondely-platform-prod, Infrastructure), not as a new image in impactpulse-prod. The cutover therefore moves sondely.com and the other production hosts from impactpulse-prod to sondely-platform-prod, and the rollback moves them back; the steps below are written that way.

Switchover checklist

Chip's steps on the day, short form. The full sequence with its go/no-go checks is Cutover day; this list says what can be done with what exists on 8 November 2026.

1. Announce the window to the organizations on Sondely, in Eastern time, with the expected length.
2. Freeze writes on the Python app. No freeze exists today: the WRITE_FREEZE switch described in Write freeze is built in neither the Python app, the new app nor catalyzeup-devops, and Container Apps has no setting that stops writes but keeps reads. What exists is taking impactpulse-prod out of service for the window, by deactivating its active revision (az containerapp revision deactivate, not rehearsed). That stops reads as well as writes, and the Python reminder loop with them: sondely.com, www.sondely.com, impactpulse.catalyzeup.ai and hit.wehappers.org answer nothing until step 4. Building WRITE_FREEZE first keeps the site readable during the window instead.
3. Migrate again, then verify, against impactpulse_prod, with the commands of scripts/migrate/README.md in CatalyzeUp/sondely-platform:

   set +x
   export ADMIN_DATABASE_URL="$(az keyvault secret show --vault-name kv-catalyzeup --name admin-database-url-sondely-prod --query value -o tsv)"
   export MONGO_URL="$(az cosmosdb keys list --resource-group rg-catalyzeup --name cosmos-catalyzeup --type connection-strings --query 'connectionStrings[?description == `Primary Read-Only MongoDB Connection String`].connectionString | [0]' -o tsv)"
   export MONGO_DB=impactpulse_prod
   npm run migrate:dry-run
   npm run migrate -- --snapshot-ref cutover-YYYY-MM-DD
   npm run migrate:verify
   

   Empty sondely_prod first (sequence step 5): the rerun is idempotent, but an upsert never removes a row deleted from Cosmos since the rehearsal, and the verifier blocks on such rows. Go only on a go verdict.
4. Move the hosts. In catalyzeup-devops azure/main.tf, point cloudflare_record.sondely_apex and sondely_www (and their sondely_apex_asuid and sondely_www_asuid TXT records) from module.impactpulse_prod to module.sondely_platform_prod, and terraform apply. Then delete the sondely.com and www.sondely.com hostname bindings from impactpulse-prod and bind them, with managed certificates, on sondely-platform-prod, as platform.sondely.com was bound (bindings are made outside Terraform). In azure/sondely-platform.tf set APP_BASE_URL and NEXT_PUBLIC_BASE_URL on sondely-platform-prod to https://sondely.com and apply. impactpulse.catalyzeup.ai (the WeHappers map calls it) and hit.wehappers.org (the HIT survey link) are also bound to impactpulse-prod today and move in the same step (WeHappers); hit.wehappers.org has no record in azure/main.tf.
5. Turn reminders on: set sondely_prod_reminders_enabled = true in azure/sondely-platform.tf and terraform apply. This creates sondely-reminders-prod, which calls https://platform.sondely.com/api/v1/jobs/reminders every 15 minutes, so platform.sondely.com stays bound.
6. Smoke test on sondely.com: /health answers status: ok and database: ok with the release commit, sign in, take a survey, ask Sona by text, call Sona by voice.
7. Rollback, if a step from 4 on fails: point the four records back to module.impactpulse_prod, move the hostname bindings back, set sondely_prod_reminders_enabled = false again, and reactivate the Python revision. impactpulse-prod still reads Cosmos, which the migration never wrote to, so it resumes on the pre-cutover data. Writes made in Postgres after step 4 follow Writes made after cutover.

The migration script

scripts/migrate/migrate-from-mongo.ts in sondely-platform, run from the repository root with npm run migrate:dry-run, npm run migrate and npm run migrate:verify, from an Azure address that the Postgres firewall rule for Azure services admits (see Infrastructure). The commands and how to read the credentials are in the script's scripts/migrate/README.md.

Not built yet, though this runbook relies on them: the --rehearsal flag (refusing the production database), --only <collection>, the reviewed merge list file for email collisions (migration-merges.json), the read-by-read diff crawler, the Python score recompute, and snapshot creation. Each is named where the runbook uses it.

Design rules

Transforms and quarantine rules

Field mapping per collection is on Data Model. The rules for what does not fit:

Go/no-go on quarantine: no ERROR row may be unresolved at cutover. Each ERROR from the production rehearsal gets a resolution recorded on the row (resolution, for example "orphan of deleted org X, accepted as lost" or "org restored from survey creator") and accepted by Chip. The final run must produce the same ERROR set, by code and source id, as the accepted rehearsal, minus resolved ones; anything new stops the cutover.

Email case collisions

The Mongo index on users.email is unique and case-sensitive, so Ana@x.org and ana@x.org can both exist if an older write path did not lower-case. Every current write path does (auth/routes.py lines 133, 312, 541, 690, 779), so collisions can only be historical.

1. The script groups users by lower(btrim(email)) before writing any user.
2. A group of one is lower-cased and migrated.
3. A group of two or more is held: every user in it, and every row that points to any of them (memberships, creator, inviter, assigner and translation approver references), goes to quarantine as EMAIL_CASE_COLLISION, with the group in detail. The script as built merges nobody.
4. Resolution is planned as a reviewed mapping file, migration-merges.json: for each group, the surviving user id and the ids it absorbs. Default proposal: the user with a password or an external id, then the most memberships, then the oldest. Merging two accounts is Chip's decision per group, because it joins two people's histories. Not built yet; the production dry run of 8 November 2026 found no collision.
5. With the mapping present, the script would rewrite foreign keys of the absorbed ids to the survivor and record the merge in MigrationRun.counts. A collision without a mapping blocks the cutover.

respondent_email on responses is not normalized: Python stores it as typed and compares it exactly in the cooldown check (surveys/routes.py line 553), so lower-casing it would change who can retake a survey.

Answers that fail the new schemas

Answers are an unchecked dictionary today (responses: Dict[str, Any]). The TS app validates new answers with a Zod schema per question type. Migrated answers are stored verbatim and validated as a report, never rewritten, because rewriting one would change a stored score and a person's history.

Each failure is a WARNING quarantine row naming the question and the reason. The report groups them by survey and reason. They do not block the cutover; they decide how strict the TS read path may be, and whether the scoring engine must tolerate them (it must, if the score recompute below is to match).

Rehearsals

Rehearsal 1: dev, for real

Run on 8 November 2026 for the data: the real Cosmos dev database migrated into sondely_dev on psql-catalyzeup-sondely-dev, then verified, with counts, hashes and recomputed scores matching and nothing unresolved. dev.sondely.com has not been switched; the new dev app serves platform-dev.sondely.com.

The dev Python app has written seed data and test documents for months; its quarantine report is expected to be noisier than production's and is the place to tune the transforms.

Rehearsal 2: a production snapshot

Run on 8 November 2026 in a different form from the plan below: read-only against impactpulse_prod itself rather than a snapshot, and written into the production database (sondely_prod on psql-catalyzeup-sondely) rather than a sondely_rehearsal database, while the Python app kept serving sondely.com. No snapshot was taken, and the read-by-read diff and the Python recompute did not run (not built).

• The dry run's first pass held 223 of 267 users as BAD_DATE (a boolean email_verified), which orphaned 157 responses and 71 assignments; the transform was fixed the same day.
• The migration then wrote 2 organizations, 267 users, 264 memberships, 1 survey with its 6 question groups and 2 translations, 24 questions, 4 sessions, 186 responses, 101 assignments and 1 API key. Nothing was held: the only quarantine rows are two STATS_DRIFT notes on the organizations.
• Verification first failed on 14 responses whose legacy WeHappers metadata, a JSON column, came back from Postgres with a float's last digit changed (27.586206896551722 written, 27.58620689655172 read). The verifier now treats such a difference as equal within a relative 1e-12 (Per-record hashes), and a second run (snapshot ref prod-rehearsal-2026-11-08b, every record unchanged) verifies go: counts, record hashes and recomputed scores.

That production copy is not the cutover. It goes stale as the Python app keeps writing to Cosmos, and the cutover empties sondely_prod and migrates again (Switchover checklist).

Data size makes repetition cheap: the WeHappers import was 257 users and 177 results (technical/happers-migration.md), and the rest of production is a few organizations (inferred; the rehearsal counts replace this guess).

Verification suite

Four checks, run in this order after every migration, each writing its result to the MigrationRun row, headless from the CLI. As built on 8 November 2026, scripts/migrate/verify.ts (npm run migrate:verify) runs the counts, the record hashes and a TypeScript score recompute; the read-by-read diff and the Python recompute are not built. The verifier writes only its report and verdict, never a migrated row.

1. Counts

For every source collection:

source documents = rows written + ERROR quarantine rows, where rows merged by a reviewed mapping count once per source document.

As built, the equation is source = primary Postgres rows + distinct ERROR-held documents + duplicate WARNING-held documents + expired verification tokens skipped: duplicates are held as WARNING and not written, and expired tokens are skipped without quarantine. Unexpected target rows, missing quarantine, changed hashes and unresolved errors each block verification.

Plus the derived tables: QuestionGroup rows equal the sum of question_groups lengths over migrated surveys minus GROUP_UNKNOWN duplicates; SurveyTranslation rows equal the distinct (survey, language) pairs found in surveys.translations and questions.translations. Collections not migrated (otp_codes, oauth_states, schema_migrations) are counted and logged, not compared.

2. Per-record hashes

For each migrated source document, a canonical hash on each side must match.

Pass: 100% of migrated documents match. As built, two records whose hashes differ still match when the only differences are non-integer numbers within a relative 1e-12, because a float in a JSON column can come back from Postgres with its last digit changed; strings, integers, booleans, nulls and structure must match exactly. The inverse mapping is code, tested both ways: inverse(transform(doc)) == shape(doc) over every fixture. Answers, score breakdowns and group results are compared as stored, which is how the hash also proves nothing in the JSON columns moved.

3. Read-by-read diff

server/scripts/verify/read-diff.ts: every GET the Python API serves, asked of both backends with the same identity, answers compared.

The crawler is read-only by construction (GET only) and also runs on cutover day, against frozen Python on Cosmos and TS on the freshly migrated Postgres.

4. Score recompute

For every response of shape A, three numbers per field (score, score_breakdown, group_results, overall_percentage):

As built, there is no Python recompute: the verifier recomputes every stored score with the TypeScript engine (src/lib/scoring) and any difference from the stored score beyond 1e-9 blocks verification, even when the hashes pass. The dev rehearsal passed that way.

Pass, as planned: TS recompute equals Python recompute for 100% of responses (floats within 1e-9). Stored versus recompute differences are expected and reported, not blocking: questions and groups can change after a response (migration 0001 rewrote groups; the WeHappers responses were scored by the import script). The report lists them by survey so a stored score is never "corrected" by accident.

Write freeze

Python has no maintenance mode today (no match for maintenance, read-only or freeze under backend/app). The freeze is a small Python change, shipped and tested before cutover:

The WeHappers map only reads, so it keeps working through the freeze.

Cutover day

Times are chosen and stated in Eastern before the day; the window is low traffic for Sondely's organizations.

Preconditions (all true the day before)

The sequence

A no-go at steps 6 to 9 means nothing has changed for users except the freeze: fix, re-run from step 5 (the target is emptied each time), or unfreeze Python and pick another day. A no-go at step 10 or 11 is a rollback.

Rollback

1. Freeze the new app (WRITE_FREEZE, to be built in the new app too), so the set of post-cutover writes stops growing.
2. Move the hosts back to impactpulse-prod, which kept running the recorded Python image with MONGODB_URL and REDIS_URL for the whole window. Cosmos is untouched (the migration only ever used the read-only key), so Python resumes on exactly the pre-cutover data.
3. Set sondely_prod_reminders_enabled = false and apply, so the new app's reminders stop before the Python app's start again.
4. Decide what happens to the writes made in Postgres after the move (below), then unfreeze Python with terraform apply.
5. Record what failed on Testing and Bugs and in the run notes. The next attempt starts again from the preconditions.

Writes made after cutover: an open decision for Chip

Every row created or changed in Postgres after the move exists only there. On rollback they are lost unless something carries them back.

Recommendation: B, with the window kept short, because it carries the data back without new code on the rollback path. To be decided with the length of the window (decision 3 on the overview).

Decommissioning Cosmos and Redis

After the rollback window closes, by Terraform in catalyzeup-devops, never by a portal click:

WeHappers

The overview listed as unknown whether WeHappers reads Sondely's database. Findings, from backend/app/main.py, the root Dockerfile (the one the deploy workflows build), catalyzeup-devops/azure/main.tf, the CatalyzeUp/wehappers.org repo and read-only Azure queries on 5 November 2026:

What the cutover must do about it:

1. Keep the two routes the map calls in the contract suite as a named scenario: the survey list with questions[].type, and paginated responses with score, responses and geo. The map finds the survey by a title containing "HIT" or "Happiness Index", so the list must stay visible to that key's organization.
2. Rotate the key before cutover, independent of decision 2: mint an org-bound READ key for the WeHappers organization, put it in kv-catalyzeup, set IMPACTPULSE_API_KEY on module.wehappers through Terraform, deploy the site, confirm the map, then revoke ip_live_wehappers_dev_key_00001. The hard-coded key sits in the source of a repo (private today) and in a seed branch of main.py; the cutover should not depend on legacy bcrypt key support for it.
3. Keep impactpulse.catalyzeup.ai served by the production app, since the site calls that host. It moves with the other hosts from impactpulse-prod to sondely-platform-prod at step 10.
4. Resolve hit.wehappers.org to the WeHappers organization in the new app (its organization-by-host lookup) before the move: the HIT survey link points there. The smoke test takes the survey on that host.
5. Do not touch the legacy Happiness Index app at happiness.wehappers.org (Vercel, its own MongoDB); retiring it is the separate plan on WeHappers Migration.