Testing and Catching Bugs
The migration replaces every line of Sondely's backend and, later, every screen. There will be bugs; the goal is that each one is caught by a test or a monitor at the cheapest moment, and that the same bug cannot come back. This page says how: the contract suite that pins today's behaviour, the harness change that runs the existing Python suite against both backends, the porting plan for the Python unit tests, the TourCockpit test pyramid Sondely adopts, the gates on every push, and the layers that catch what the tests miss, before and after cutover.
Related: Migration overview, API Routes, Business Logic, Cutover Runbook, Infrastructure, Screens into Next.js.
As built on 8 November 2026
The new app, CatalyzeUp/sondely-platform, was written from the Python code and these pages with its own tests, and the Python app's behaviour was not pinned first. What this page plans and what exists:
| Planned here | State |
|---|---|
| Phase 1 contract suite against Python and TypeScript, and the Python suite run remotely | Not built |
| Scoring tables ported as data | Built: the scoring engine's tests carry the Python cases (tests/unit/scoring/) |
| Unit, component, integration, end to end | Built: 81 unit test files, 12 component, 33 integration against Postgres (tenant isolation included), 6 Playwright specs on a desktop and a phone project in Chromium, and 2 for Sona (the drawer, and a call against a fake Voice Live server). The migration script has 9 test files of its own (tests/migrate/) |
| Coverage floors, usability capture, client error reporter | Not adopted yet |
| CI | One verify job on ubuntu-latest on every push and pull request to develop and main: lint, typecheck, unit, component, integration, build, end to end, Sona's voice call. No Discord notification yet (Infrastructure) |
| Score recompute and migration counts and hashes | Built in the migration verifier, passing on the dev rehearsal (Cutover Runbook) |
| Read-by-read diff | Not built |
Where a section below names a path under server/, read it as the same path at the root of sondely-platform.
What exists today in the Python app
Read from /home/chipdev/ideaplaces-meta/sondely at commit 06debbf.
| Layer | Where | What it covers |
|---|---|---|
| Backend tests | backend/tests/*.py, backend/tests/unit/, backend/tests/integration/ |
734 tests (708 def test_ definitions by grep; the rest are parametrized cases, inferred). In-memory database (USE_INMEMORY_DB=true), Cosmos sort guard on (COSMOS_SORT_GUARD=true) |
| Real Mongo integration | backend/tests/integration/test_mongodb_writes.py (8 tests) |
Runs against a mongo:7 service in CI, catches engine behaviour the in-memory database cannot |
| Frontend tests | 16 files under frontend/src/**/*.test.ts(x), vitest with jsdom (frontend/vitest.config.ts) |
Take flow components, catalog completeness and ICU validity, colour contrast, a few pages |
| Browser smoke | scripts/e2e_take_flow.py, scripts/e2e_export.py, scripts/e2e_localization.py |
Python Playwright, run by hand against a local stack. Not in CI |
| CI | .github/workflows/ci.yml: frontend, backend-tests, backend-integration |
Lint, tsc --noEmit, vitest, build, pytest, Mongo integration. Discord post on failure (DISCORD_WEBHOOK_IMPACTPULSE_DEV). Runs on ubuntu-latest |
| Email capture | GET and DELETE /api/v1/dev/emails, registered in backend/app/main.py only when ENVIRONMENT is development or test |
Every email the app would have sent, with its links |
The test client is built in backend/conftest.py:
async with AsyncClient(app=app, base_url="http://test") as ac:
The app object is passed in, so every test runs inside the Python process. That one line is why the suite cannot reach a TypeScript server today, and it is the line the harness change below replaces.
Phase 1: the contract suite
A new TypeScript suite in contract/ at the repo root, written with vitest, that talks to /api/v1 over HTTP and nothing else. It records what Python does today, route by route, so the TypeScript backend has a written target instead of a reading of Python code.
| Property | Rule |
|---|---|
| Target | SONDELY_BASE_URL, required. The suite never imports server code from either backend |
| What a test asserts | Status code, the response's shape as a Zod schema (strict, so an added or dropped field fails), and the values that matter (ids, counts, scores, statuses, ordering). Timestamps are compared as instants, since Python stores them without a zone and TypeScript writes timestamptz |
| Setup | Through the public API wherever possible (register, create org, create survey), and through the test-only routes below for state the API cannot create (an expired code, a response from last year, a member with a past due date) |
| Quirks | Recorded as they are, not as they should be. Where a decision changes a behaviour on purpose (the error format, decision 4 in the stack plan), the test carries the new expectation for TypeScript and the old for Python, with the decision named in the test |
| Order of writing | Respondent path first (public survey, sessions, respond, can-attempt, auto-enroll, upgrade, claim), then auth, surveys and questions, organizations and members, assignments and reminders, impact and exports, translations, API keys, notifications, geo |
| Done (phase 1) | Every route in API Routes has at least one test, and the suite is green against Python on develop |
How it runs in CI. One job, contract, with two targets:
| Target | Server | Database | Required |
|---|---|---|---|
python |
uvicorn app.main:app with ENVIRONMENT=test |
mongo:7 service, as backend-integration uses |
From phase 1 |
ts |
next start of server/ |
Postgres service, migrated with prisma migrate deploy |
From phase 2, with a ratchet |
The ratchet: until parity, the TypeScript target reads contract/pending-ts.json, a list of the route groups not yet ported. A test in a listed group may fail; a test in an unlisted group may not, and a listed group whose tests all pass fails the job until it is removed from the list. The list only gets shorter, and phase 3 is done when it is empty. Nothing is skipped silently: the job prints the pending groups on every run.
Running the Python suite against both backends
The contract suite is new. The 734 Python tests are older, larger and already describe the product. About 340 of them can run against any server over HTTP with a small harness change, which doubles the contract at almost no writing cost.
| Kind (measured) | Count | How it runs against TypeScript |
|---|---|---|
| Pure HTTP in and out | about 220 | As is, once the client takes a base URL |
HTTP, but seeds or reads the database directly (get_database() in the test), or patches a module |
about 120 | Through three test-only routes (reset, seed, read back), a fake model mode and a fake geo lookup |
| Python internals (scoring, assessments, translation validator, reminder engine, report export, migrations) | about 394 | Not run remotely. Ported or dropped, see the next section |
Files that read the database most (grep of get_database and direct collection calls): test_drip_campaigns.py, test_migration.py, test_backfill_geo.py, test_tenant_assignments.py, test_translation_edge_cases.py, test_response_context.py, test_oauth_return_origin.py, test_tenant_membership.py, test_org_counts_and_enroll.py. Twelve files use patch or monkeypatch.
The harness change
| Piece | Change |
|---|---|
backend/conftest.py, async_client |
When SONDELY_BASE_URL is set, AsyncClient(base_url=os.environ["SONDELY_BASE_URL"]) with no app. When it is not set, today's in-process client, unchanged |
backend/conftest.py, client |
The same switch for the sync TestClient(app) used by eight files (test_api_keys.py, test_export_endpoints.py, test_impact.py, test_custom_domain.py, test_notifications.py, test_oauth_return_origin.py, test_org_counts_and_enroll.py, test_templates.py): an httpx.Client on the base URL |
backend/conftest.py, clean_db |
In remote mode, POST /api/v1/test/reset instead of deleting collections in process |
backend/pytest.ini |
A new marker, inprocess: needs the Python app in this process. It must be registered because --strict-markers is on |
| Tests that read Python internals | Marked @pytest.mark.inprocess. A remote run is SONDELY_BASE_URL=... pytest -m "not inprocess". The marker count is printed and only goes down |
| Tests that seed or read the database | Rewritten to call the test-only routes, one file at a time. Each file that moves loses its inprocess marker |
The test-only routes
Both backends expose the same routes under one test prefix, defined in API Routes. They exist only when ENVIRONMENT=test and answer 404 in every other environment, the way TourCockpit's src/app/api/auth/test-login/route.ts answers 404 without its secret, and a unit test asserts the 404 for each one. Today's email capture (GET and DELETE /api/v1/dev/emails, backend/app/main.py) moves under the same prefix.
| Route | Does | Used by |
|---|---|---|
POST /api/v1/test/reset |
Truncates every table (or collection, on Python) | clean_db, the start of every contract file |
POST /api/v1/test/seed |
Seeds a named fixture with overrides (an org with owners, members and respondents, surveys, responses), including states the API refuses to create: an expired code, a response from last year, a member past their due date | Tests that today insert into Mongo directly |
GET /api/v1/test/read/{model}/{id} |
Returns one stored row | Tests that assert what was written, not what was answered (a hashed key, a stored score breakdown, a reminder count) |
GET, DELETE /api/v1/test/emails |
The intercepted emails, with their links | Verification, invitation, one-time code and reminder tests |
POST /api/v1/test/sign-in |
A session for a seeded user | The Playwright suite, as TourCockpit's test-login seam |
model is a fixed list, so the route cannot read arbitrary storage. The reminder tests in test_drip_campaigns.py also drive the reminder engine directly; whether they need a fourth route (run one reminder tick at a given time) or move to the scheduled job's own tests is inferred and decided in Business Logic.
The fakes
| Fake | Switch | Behaviour |
|---|---|---|
| Model | An environment switch read when ENVIRONMENT=test (name set in API Routes) |
Survey translation and UI translation return a deterministic pseudo translation ([fr] prefixed text, ids and counts intact), the same idea as scripts/translate_ui.py --offline. A test can ask for a broken answer (a dropped question, a changed type) to prove the validator refuses it |
| Geo | Same, a test-mode switch | /api/v1/geo/detect reads a fixed table keyed by the X-Forwarded-For address (a Canadian, a Quebec, an unknown and a private address), instead of the bundled GeoLite2 database (backend/app/api/geo/routes.py) |
| Already exists | Captured in test mode, read and cleared through the email route above |
Porting the Python unit tests
The roughly 394 tests of Python internals do not run against TypeScript. Each has one of three fates.
| Group | Files | Count | Fate |
|---|---|---|---|
| Scoring and assessments, table driven | test_scoring.py (27), test_scoring_comprehensive.py (38), test_structured_assessments.py (53), plus the scoring and aggregation cases in test_impact.py and test_report_export.py (which of those cases count is inferred) |
about 190 | Rewritten in TypeScript before cutover. Each table becomes a vitest table over the TypeScript scoring engine with the same inputs and expected outputs, copied verbatim. A sample of each table also runs through HTTP in the contract suite, so both backends answer it. Scoring is the logic where a quiet difference does the most harm |
| Migrations and backfills | test_migration.py (41), test_backfill_geo.py (8), and the WeHappers migration cases |
about 60 | Dropped. One-off code that does not exist in the new backend. The data migration script gets its own tests (see Data Model) |
| The rest of the internals | test_translation_service.py, test_translation_edge_cases.py, test_ui_translation.py, test_email_localization.py, test_group_color.py, test_cosmos_sort_guard.py, unit/test_*_model.py |
about 144 | Ported by behaviour, not by line. The translation validator, email locale resolution and model validation get TypeScript unit tests written against the behaviour the Python tests describe. Cosmos-specific guards are dropped with Cosmos |
The table-driven cases are copied as data, never re-derived: a person reading the Python test and rewriting the expected value from the code would copy the bug along with it.
The TourCockpit test pyramid, adopted
Copied from ideaplaces-tour-platform at 74e9a2a53 (the plan read it at f952d0eea), with the header naming the source file, as Framework Copy describes.
| Layer | TourCockpit source | In Sondely |
|---|---|---|
| Unit | vitest.config.ts: node environment, tests/unit/**/*.test.ts, server-only stubbed, ACS_CONNECTION_STRING="" so no test can send real email |
Same file. Scoring, validators, translation checks, source scans |
| Component | vitest.component.config.ts: happy-dom, tests/component/**/*.test.tsx, tests/component/setup.ts |
Same file. Every screen that moves brings its component tests |
| Integration | vitest.integration.config.ts: real Postgres, tests/integration/global-setup.ts, per-worker-db.ts (a database per worker copied from the migrated template), setup.ts (truncate per test), 30 second test and hook budgets, tests/integration/helpers.ts (seedTenant, asUser) |
Same files, without the call and migration parts in tests/integration/_parts.ts. Services and route handlers against a real database |
| End to end | playwright.config.ts: production build (next start), its own database, zero retries, eight workers, desktop and mobile (Pixel 7, files named *.mobile.spec.ts) projects, --keepAliveTimeout 65000, sign-in through the gated /api/auth/test-login seam (Sondely: /api/v1/test/sign-in), helpers in tests/e2e/_auth.ts |
Same config, stripped of the assistant, WhatsApp and pay variables |
| Usability | tests/visual/capture-usability.mjs, tests/visual/usability-check.ts, tests/visual/rubric.md, tests/visual/contrast.mjs; method in the Usability Verification Playbook |
Sondely already has frontend/tests/visual/contrast.mjs and rubric.md; the capture and the gate are added. Journeys at phone, tablet and desktop |
| Coverage | vitest.coverage.ts (v8, every src file counted, data excluded), scripts/coverage-merge.ts, scripts/coverage-changed.ts, coverage-floors.json |
Same scripts |
Coverage floors
TourCockpit's coverage-floors.json, adopted as Sondely's starting floor, because server/ starts empty and every file arrives with its tests:
| Scope | Floor |
|---|---|
Whole server/src |
85% |
| Backend core | 95% |
| Backend other | 90% |
| Route handlers | 90% |
| Server actions | 90% |
| Components | 80% |
Changed lines in a pull request (CHANGED_LINES_FLOOR in scripts/coverage-changed-lib.ts) |
90% |
The Vite app under frontend/ is outside the floor until it is deleted.
Gates copied as tests
| Gate | TourCockpit file | What it stops |
|---|---|---|
| Tenant isolation | tests/integration/tenant-isolation.test.ts |
Org A reading, editing or deleting a record of org B by id. Every attempt must come back as not found, not an empty list. Rewritten for surveys, responses, members, assignments and API keys |
| Tenant model coverage | tests/integration/tenant-coverage.test.ts |
A Prisma model with organizationId that is missing from the tenant model list, which would be an unscoped hole. Every child table carries organizationId (Data Model) |
| Raw SQL | tests/unit/no-raw-tenant-sql.test.ts |
A raw query on a tenant table that skips the tenant filter |
| Row-level security | tests/integration/rls.test.ts, rls-auth-carveout.test.ts |
Only if the data model adopts TourCockpit's constrained database role (decided in Data Model) |
| Refusal sites | tests/unit/refusal-sites.test.ts with refusal-sites.json, tests/integration/refusal-probes.test.ts |
A new refusal added without its class being read. Adopted with coded errors (phase 5, or at cutover if decision 4 says so) |
| UI and API parity | tests/unit/ui-api-parity.test.ts, ui-read-parity.test.ts |
A screen that can do something the API cannot, which an agent then cannot do either. Adopted when screens start moving |
| Translation | tests/unit/i18n-*.test.ts |
See Translation |
CI gates per push
The target workflow as planned. The new app's CI lives in its own repository, so the Python and Vite jobs stay in CatalyzeUp/sondely until those apps are retired. Runners are GitHub-hosted, ubuntu-latest (decision 4, Infrastructure). What runs today is in As built above.
| Job | Runs | Blocks merge |
|---|---|---|
| Static | npm run typecheck, npm run lint, the no-attribution scan, in parallel as scripts/ci/parallel-checks.sh does |
Yes |
| Unit | npm run test:unit, including the i18n and source-scan gates |
Yes |
| Component | npm run test:component |
Yes |
| Integration | npm run test:integration against Postgres, including tenant isolation |
Yes |
| Contract | The contract suite against Python and against TypeScript, with the ratchet | Yes |
| Python remote | The Python suite with SONDELY_BASE_URL pointed at the TypeScript server, -m "not inprocess" |
From phase 3 |
| Python | Today's backend-tests and backend-integration |
Until backend/ is deleted |
| Frontend | Today's frontend job |
Until the Vite app is deleted |
| End to end | npm run test:e2e, desktop and mobile, production build |
Yes |
| Migration rehearsal | A copy of the production schema migrated forward, as TourCockpit's scripts/ci/migration-rehearsal.sh |
On pull requests that carry a migration |
| Coverage | coverage:merge against the floors, then coverage:changed |
Yes |
| Usability | npm run usability:check over the committed grades |
Yes, once the first screen has moved |
| Discord | Posts on red. Today a curl to DISCORD_WEBHOOK_IMPACTPULSE_DEV per job; the Ideaplaces/discord-ci-status action (deletes the post when green) if it is reachable from the CatalyzeUp org, which is inferred |
Not a gate |
How bugs are caught and handled
The rule
Every bug found becomes a failing test first. Before the fix, a test that reproduces it is written in the lowest layer that can hold it (a scoring case in a unit table, a route behaviour in the contract suite, a screen in Playwright) and seen to fail. Then the fix. Then the test passes, and the bug is a regression the suite remembers. A bug found in the read-by-read diff or in production follows the same rule: the diff or the log line is the evidence, the test is the fix's first commit.
Before cutover
| Layer | Catches | Where it runs |
|---|---|---|
| Unit and component tests | Logic and rendering mistakes in what was just written | Every push |
| Integration tests | Service and database mistakes, tenant leaks | Every push |
| Contract suite | A TypeScript route that answers differently from Python: status, shape, values, ordering | Every push, both targets |
| Python suite, remote | The same, through about 340 existing tests that already encode the edge cases | Every push from phase 3 |
| Read-by-read diff | What the tests did not think to ask. Every GET the API serves is replayed against both backends on the same production snapshot (Cosmos for Python, the migrated Postgres for TypeScript), and every difference is listed. Phase 3 is not done until the list is empty or every line on it is a decided difference |
On a production snapshot, before the freeze |
| Score recompute | Every stored response's score recomputed by the TypeScript engine and compared with the score Python stored | On the snapshot, and again after the final migrate |
| Migration counts and hashes | A record lost, changed or quarantined in the move | Every rehearsal and the real run (Cutover Runbook) |
| Rehearsals | A runbook step that does not work as written | On dev, then on a production snapshot, as many times as it takes to run clean |
| End to end and usability | A screen that works in a component test and fails in a browser or on a phone | Every push, and per screen as screens move |
After cutover
| Layer | What it does | State today |
|---|---|---|
| Structured logs | Every server log line is one JSON object with level and message (src/lib/logger.ts in TourCockpit, copied). Every request logs its route, org, status and duration |
Python already logs JSON (backend/app/core/logging.py) |
| Client error reporter | Uncaught errors, unhandled rejections and console.error in the browser are posted to /api/client-log and land in the same stream (src/lib/errorReporter.ts, src/app/api/client-log/route.ts, src/app/error.tsx, src/app/global-error.tsx) |
Sondely has none; the Vite app reports nothing. Copied in phase 2, and wired into the Vite app too so the screens not yet moved report as well |
| Health check | GET /health runs SELECT 1 through the app's constrained role, names the deployed commit, and answers 503 when the database is unreachable. The deploy waits on it |
Built in the new app; Python has /health without a database check. Container app probes are not set yet |
| Discord CI alerts | A red build on any branch posts to the team channel | Exists, kept |
| C3 self-healing monitor | Errors are read from Log Analytics every five minutes, summarised to Discord, and a C3 agent investigates each alert and opens a suggested fix pull request | Not wired for Sondely. The IdeaPlaces pipeline (func-alert-summarizer-production, read by the alerts-production trigger in ~/.c3/triggers.json) reads law-ideaplaces-docs; Sondely's container apps log to law-catalyzeup (catalyzeup-devops/azure/main.tf). No trigger in ~/.c3/triggers.json names Sondely or CatalyzeUp. Wiring it is a phase 2 item in Infrastructure and must be live before cutover |
| Deploy monitor | The deploy-monitor agent watches each push's CI and deploy, then the health URL |
Kept |
Rollback triggers
The rollback is moving the hosts back to the Python container app, impactpulse-prod, with Cosmos untouched (Cutover Runbook). During the rollback window, any of these calls it. The call is Chip's; this list is what he is shown.
| Trigger | Seen in |
|---|---|
/health failing for more than a few consecutive probes |
Container app health, deploy monitor |
| A respondent cannot load a survey or submit a response (any 5xx on the public survey, session or respond routes) | Logs, C3 monitor |
| Sign-in failing for existing users (password, Google, Apple or one-time code) | Logs, smoke test |
| A score shown that differs from the score Python stored for the same response | Score recompute, a customer report |
| Any sign of data from one org visible to another | Anyone, any channel. Rollback first, investigate after |
| Emails not sent (verification, invitation, reminders) | Logs, the reminder job's own log line |
| Migration counts or hashes that do not match | The cutover run's log |
A bug that does not meet a trigger is fixed forward: failing test, fix, push, deploy.
Not yet covered
- Whether the contract suite should also cover the WeHappers paths, which depends on whether WeHappers reads Sondely's database (open in the stack plan).
- PDF exports: compared by text content and page count, or byte for byte. Open in Business Logic.
- The Postgres service for the
tscontract target, if the contract suite is built (Infrastructure).