Introduction

The Exam Platform is a group of services involved in the creation, editing, and serving of freeCodeCamp certified exams.

Get started with the architecture of the platfrom.

Apps

This platform consists of the following applications:

Auth API

Candidate identity and the attempt path. The only application candidate devices reach.

cp .env.example .env
cargo run -p auth-api
docker compose up --build

Curriculum Database

Self-hosted Dolt: exam content and its whole history. MySQL wire protocol.

Only root exists. App roles for eDd and aAPI are not set up yet: roles.

Setup

cd databases/curriculum
cp .env.example .env   # CDB_ROOT_PASSWORD=$(openssl rand -hex 32)
docker compose up -d --wait

scripts/server.sh replaces the image’s entrypoint, so restarts never create users, run init files or migrate. sql-server creates root from CDB_ROOT_PASSWORD only when no privileges file exists. To rotate, run ALTER USER 'root'@'%' IDENTIFIED BY '<new>', then update CDB_ROOT_PASSWORD and recreate the container. If the variable is changed without the ALTER USER, the healthcheck fails.

SQL shell, through the container’s dolt CLI. --use-db takes curriculum/<branch> or curriculum/<commit hash>:

cdb() { docker compose exec -T curriculum-db sh -c 'DOLT_CLI_PASSWORD="$DOLT_ROOT_PASSWORD" exec dolt --host 127.0.0.1 --port 3306 --user root --no-tls "$@"' cdb "$@"; }
cdb sql -q 'SELECT DOLT_VERSION()'

Once-off

Create the database with dev branch, then migrate it. cargo test -p cdb uses the same URL, and creates and drops a database per test:

cdb sql -q 'CREATE DATABASE curriculum'
set -a; . ./.env; set +a
export CDB_MIGRATOR_URL="mysql://root:$CDB_ROOT_PASSWORD@127.0.0.1:13306/curriculum"
cargo run -p cdb --bin cdb-migrate

Config

dolt_transaction_commit is disabled as this produces a noisy history.

Dolt

A session’s branch is session state. dolt_checkout() stays on a pooled connection for the next borrower, and ROLLBACK does not undo it: BEGIN; CALL dolt_checkout('x'); ROLLBACK; leaves ACTIVE_BRANCH() on x. Never rely on the branch a pooled connection happens to be on. Either:

a. Start every transaction by naming its branch. USE a revision database, which reads, writes, and dolt_commits on that branch without touching checkout state:

BEGIN; USE `curriculum/<branch>`; ... COMMIT;

b. Give each branch its own pool, whose connections run that USE on connect (sqlx after_connect) and never call dolt_checkout.

Table names can also be branch-qualified: `curriculum/prd`.exams. Only cdb-migrate calls dolt_checkout, on a connection it opens and closes itself.

Dolt does not support SELECT FOR UPDATE.

Migrations

cdb-migrate (CDB_MIGRATOR_URL) applies migrations/ with sqlx, as edb-migrate does for eDb, with these differences for Dolt:

Dolt-specific notes:

• Only dev receives the migration
  • Branches get the schema when dev is merged into them. Never independently run migrations on two branches.
• One Dolt commit per migration
  • Message cdb-migrate: NNNN <description>, author cdb-migrate.
  • Each commit holds the schema change and its _sqlx_migrations row, so diffing, reverting, and cherry-picking a migration is one commit.
• Applied on a scratch branch
  • Each run, under a GET_LOCK named for the database:
  1. Recreates branch cdb-migrate from dev’s HEAD and checks it out.
  2. Applies each pending migration there, then CALL DOLT_COMMIT('-A', ...).
  3. Merges cdb-migrate into dev: a fast-forward, or a merge commit (cdb-migrate: merge NNNN..MMMM) if dev moved meanwhile. A conflict aborts the merge and fails the run.
  4. Deletes cdb-migrate.
• _sqlx_migrations table is versioned
  • Not in dolt_ignore.
  • Otherwise, sqlx would not see applied migrations on other branches, nor after a merge.

The scratch branch is needed because:

• Dolt commits the SQL transaction on DDL, so a migration is never atomic.
• dolt_commit -A on dev would sweep in whatever uncommitted writes other sessions left in dev’s working set.
• dolt_commit inside an open transaction can commit a stale snapshot, so it always runs after the migration’s transaction.

On the scratch branch, a failed or interrupted run leaves dev untouched. The next run recreates the branch and starts over. A failed run leaves cdb-migrate behind for inspection, with --use-db curriculum/cdb-migrate.

Only eDd runs cdb-migrate, before each of its deploys, after eDb’s migrate. aAPI only reads cDb, from branches whose schema changes only on a merge from dev. Running it there gains nothing and would give aAPI’s stack DDL credentials.

TODO: merging dev into stg/prd also brings unpromoted content and manifest rows. Cherry-picking the per-migration commits is an option.

GC and Deletion

Automatic GC is on (config/server.yaml). GC runs in the background as garbage builds up, including from each cdb-migrate run’s deleted scratch branch.

Never disable automatic GC without a replacement. With it off, every CALL DOLT_GC() breaks all open connections, fails in-flight queries, and leaves the calling connection unusable. Apps would then need to retry and reconnect, and GC would need maintenance windows.

Never use docker compose down -v, docker volume prune or hard resets on real data.

Desktop App

Serves exam content to the candidate and observes the station.

Linux needs the Tauri prerequisites (webkit2gtk-4.1, libsoup-3).

cp .env.example .env
bun run tauri:dev
bun run tauri:build
bun tauri icon public/logo.svg -o backend/icons

Examiner Dashboard

Where examiners author, review, promote, and moderate.

The Dockerfile builds client/ with Bun, server/ with cargo, and copies dist to /app/web.

client/src/layouts/base.astro is the HTML shell. Each pages/*.astro is routing only and mounts exactly one views/*.tsx React component, which wraps itself in components/app-root.tsx (React providers).

cp .env.example .env
cargo run -p examiner-dashboard
bun --filter examiner-dashboard-client dev
docker compose up --build

Examiner Database

Hosted (eventually) Postgres database.

Roles

Bootstrap creates the roles and schemas, and revokes TEMP from PUBLIC. Migrations create every table and grant the app roles access to it explicitly: no default privileges, so each new table is a deliberate decision. Neither app role can read edb - the migration history. Queries should name the schema (e.g. edd.users), and should never rely on unqualified search_path.

edb-migrate (env EDB_MIGRATOR_URL) holds a Postgres advisory lock, so concurrent runs from both backends’ releases apply each migration once. It waits at most 60s for a lock (edb::LOCK_TIMEOUT) and 300s overall, then exits non-zero. It verifies the checksums of the migrations it embeds, and ignores applied migrations it does not: an older release runs against a newer schema. Each migration runs in a transaction: failed ones roll back, earlier ones stay applied.

Deploys run the edb-migrate bin from the image being deployed, before replacing the app, as the one-shot compose service migrate. The apps never migrate or check schema versions.

For hosted Postgres: as an admin role, run initdb/10-bootstrap.sql, then ALTER ROLE <role> PASSWORD '...' for edb_migrator, edd, and aapi.

Schemas

Schemas are organizational. They are not ownership or access boundaries. edb_migrator owns every table, and grants decide what each app reads and writes.

Place a table alongside its writers, not its readers: one writer goes in that app’s schema, two go in shared. Moving a table between schemas later is a rename, so it follows the add, release, remove rule.

Bootstrap is idempotent and creates shared like the others: CREATE SCHEMA IF NOT EXISTS shared AUTHORIZATION edb_migrator. It cannot be a migration, as edb_migrator has no CREATE on the database (DDL boundary).

Foreign keys may cross schemas (e.g. shared.x.candidate_id REFERENCES aapi.candidates (id)). edb_migrator owns both ends, so no REFERENCES grant is needed, and RI checks run as the owner, so the writing app needs no SELECT on the referenced table. A cross-schema FK still couples the apps:

• Deliberately choose ON DELETE. With the default, the referenced app’s deletes start failing once the other app’s rows point at them, and its running image may not expect that. Adding the FK must work with both running images and their rollbacks.
• On a populated table, add it as NOT VALID, then VALIDATE CONSTRAINT in a later migration: validation scans the table, and the add locks both tables.
• Dropping the referenced table or column needs the FK dropped first (in an earlier release).

Setup

Needs sqlx-cli 0.9: cargo install sqlx-cli --version 0.9.0 --locked --no-default-features --features rustls,postgres,sqlx-toml.

cp .env.example .env   # set the four passwords, and the same in the two URLs
docker compose up -d --wait
set -a; source .env; set +a
cargo run -p edb --bin edb-migrate
git config core.hooksPath .githooks   # pre-commit refreshes and stages .sqlx/

Start either app. initdb/ should only be run on an empty volume. Changing or resetting: docker compose down -v, then the steps above.

Changing the schema

From this directory:

1. cargo sqlx migrate add -r -s <name>.
2. Write the up and down SQL, naming the schema. Pick the schema by who writes the table (schemas). down is for local development only; production never runs it.
3. Grant each app role exactly what it needs on the new table, in the same migration, e.g. GRANT SELECT, INSERT, UPDATE, DELETE ON edd.moderation_outcomes TO edd and GRANT SELECT ON edd.moderation_outcomes TO aapi. A new schema also needs GRANT USAGE.
4. cargo run -p edb --bin edb-migrate (or cargo sqlx migrate run). Iterate on an uncommitted migration with cargo sqlx migrate revert.
5. Change the queries, then run cargo sqlx prepare with eDb running: builds are offline (.cargo/config.toml), so a new or changed query fails to compile until .sqlx/ has it. The pre-commit hook also runs it and stages .sqlx/. It never blocks the commit, so check its warning; CI’s offline clippy fails on a stale .sqlx/.
6. Tests: DATABASE_URL=$EDB_ADMIN_URL cargo test -p edb. Use #[sqlx::test(migrations = false)] and call testing::setup first: it bootstraps, then migrates as edb_migrator. Query through testing::database_as(&pool, Role::Edd) to test as the app does; test denials with SET LOCAL ROLE.

Rules:

• Only add, release, then remove: drop or rename only once no running or previous image of eDd or aAPI uses the object. Each migration must work with both the running images and their rollback images. Nothing checks this.
• Either backend’s release applies every migration in its build. Both apps embed the crate, so both release on any change here; whichever deploys first migrates.
• Code that needs a new migration can ship with it: the deploy migrates before starting the app. The other app’s running image must still work with the new schema.
• Never edit an applied migration: sqlx checksums each one, and edb-migrate fails, so no backend deploys.
• A failed migration blocks every backend deploy until a release fixes it. Schema changes are never rolled back automatically.

Web App

The candidate’s entry surface, and second device to potentially proctor.

cp .env.example .env
bun --filter web-app dev
bun --filter web-app build
docker compose up --build # reads AUTH_API_URL from .env at runtime

Architecture

Architecture

graph LR
  subgraph candidate[Candidate devices]
    dApp["Desktop App<br/>[dApp]"]
    wApp["Web App<br/>[wApp]"]
  end

  aAPI["Auth API<br/>[aAPI]"]
  cDb[("Curriculum Database<br/>[cDb]")]
  eDd["Examiner Dashboard<br/>[eDd]"]
  eDb[("Examiner Database<br/>[eDb]")]

  dApp <--> aAPI
  wApp <--> aAPI
  aAPI --> cDb
  aAPI <--> eDb
  eDd <--> cDb
  eDd <--> eDb

eDb is used by eDd and aAPI, one schema each (edd, aapi) plus shared for tables that both write. All are owned by role edb_migrator; apps connect as runtime roles that cannot change the schema.

Neither app migrates. During a deployment of either eDd or aAPI, Komodo stands the migrate service up which runs the edb-migrate binary, then, if successful, either app is started.

Auth

All service-to-service traffic stays on the private platform network. Each database grants per-service users with least privilege.

Examiner → eDd

graph LR
  examiner(["Examiner"]) -->|"OIDC SSO login, session cookie; authZ by roles and scopes in eDb, enforced by eDd server"| eDd["eDd"]

dApp → aAPI

graph LR
  dApp["dApp"] -->|"Candidate login, short-lived bearer session token; authZ scoped to candidate's own attempt"| aAPI["aAPI"]

wApp → aAPI

graph LR
  wApp["wApp"] -->|"One-time pairing code from dApp attempt, short-lived bearer session token; authZ scoped to paired attempt"| aAPI["aAPI"]

aAPI → eDb

graph LR
  aAPI["aAPI"] -->|"DB role aapi: no DDL; authZ read/write candidates, attempts, responses, evidence"| eDb[("eDb")]

eDd → eDb

graph LR
  eDd["eDd"] -->|"DB role edd: no DDL; authZ examiner identity, sessions, moderation outcomes; read candidate data"| eDb[("eDb")]

Contributing

First Time Setup

Prerequisites

# For working with database stuffs
cargo install sqlx-cli --version 0.9.0 --locked --no-default-features --features rustls,postgres,sqlx-toml
# For working on docs/
cargo install mdbook@0.5.4 mdbook-mermaid@0.17.1 --locked # optional

Clone

git clone https://github.com/freeCodeCamp/exam-platform.git
cd exam-platform
bun install
docker network create platform

bun install also installs the Husky pre-commit hook.

Environment

Copy every example, then fill blanks:

for d in apps/auth-api apps/examiner-dashboard apps/web-app apps/desktop-app databases/examiner; do
  cp -n "$d/.env.example" "$d/.env"
done

.env is read automatically only by:

Rust apps read only the process environment. Export the app’s .env into the shell before cargo:

set -a; source apps/auth-api/.env; set +a
cargo run -p auth-api

Or use direnv: echo dotenv > .envrc && direnv allow in each app directory, then run cargo from there. .envrc is not in .gitignore; do not commit it.

All variables: Environment Variables.

eDb

cd databases/examiner
docker compose up -d --wait
set -a; source .env; set +a
cargo run -p edb --bin edb-migrate

Run edb-migrate again after pulling new migrations. Reset with docker compose down -v.

Editor

VS Code: install recommended extensions (.vscode/extensions.json). Settings run clippy on save, set ENVIRONMENT=development for rust-analyzer, and link both Cargo workspaces.

Development

Run

cargo run rows need their .env exported first (Environment). bun run dev starts wApp and eDd client together. eDd client proxies /api to eDd server. wApp and dApp call aAPI at AUTH_API_URL.

Images: docker compose -f apps/<app>/compose.yaml up --build. Build context is repo root.

Workspaces

Shared versions live in package.json catalogs: reference with catalog: or catalog:astro, do not pin in members.

Check

CI (.github/workflows/ci.yml) runs only jobs whose paths changed. Run the same locally before pushing:

# Root Cargo workspace (eDb running; source databases/examiner/.env for EDB_ADMIN_URL)
cargo fmt --all --check
cargo clippy --workspace --all-targets --locked
cargo clippy -p runtime --no-default-features --all-targets --locked
DATABASE_URL=$EDB_ADMIN_URL cargo test --workspace --locked

# dApp backend
cargo fmt --all --check --manifest-path apps/desktop-app/Cargo.toml
cargo clippy --workspace --all-targets --locked --manifest-path apps/desktop-app/Cargo.toml
cargo test --workspace --locked --manifest-path apps/desktop-app/Cargo.toml

# Bun
bun run check
bun run build

# Deploy Action (migration gate, rollback)
(cd deploy && bun test)

Clippy all is deny. Format Rust with rustfmt (rustfmt.toml), eDd client with bun --filter examiner-dashboard-client fmt (oxfmt).

Database Changes

eDb schema, migrations and queries live in crate edb (databases/examiner/). Run every step from that directory with eDb up (docker compose up -d --wait). Only add, release, then remove: drop or rename only once no running or previous eDd or aAPI image uses the object. Never edit an applied migration.

sqlx-cli reads EDB_MIGRATOR_URL from its .env; cargo does not, so export it first.

1. Create a reversible migration pair:

cargo sqlx migrate add -r -s <name> # migrations/NNNN_<name>.{up,down}.sql

2. Write up and down SQL, naming the schema (edd.users; shared if both apps write the table, see eDb schemas), and grant each app role what it needs on new tables. down is for local development only; production never runs it.
3. Apply it to local eDb:

cargo run -p edb --bin edb-migrate # or: cargo sqlx migrate run

To iterate on uncommitted migration: cargo sqlx migrate revert, make edits, then run it again.

4. Add or change queries in src/edd/, src/aapi/ or src/shared/ (one module per table group, each an impl Database block of query!/query_as!). SQLX_OFFLINE is true in .cargo/config.toml, so cargo check -p edb checks them against committed .sqlx/, and fails on a new or changed query. To validate against local eDb, override it: SQLX_OFFLINE=false cargo check -p edb.
5. Test with #[sqlx::test(migrations = false)], calling testing::setup first: it bootstraps roles and schemas, then migrates as edb_migrator. Query through testing::database_as to run as the app’s role. #[sqlx::test] reads DATABASE_URL, not sqlx.toml, and creates a database per test:

DATABASE_URL=$EDB_ADMIN_URL cargo test -p edb

6. Refresh offline query data:

cargo sqlx prepare

The Husky pre-commit hook runs this and stages .sqlx/ when eDb files are staged. This step never blocks the commit: if it warns with cargo sqlx prepare failed; .sqlx/ may be stale, manually run step 6.

7. Commit migration, queries and .sqlx/ together. Local (.cargo/config.toml), Docker and CI builds all use SQLX_OFFLINE=true. CI fails on a stale .sqlx/: offline clippy on a missing query, cargo sqlx prepare --check on one whose types changed.

Commits and PRs

Conventional Commits: release-please reads them to version each app (Releases).

<type>(<scope>): <summary>

Follow Database and Rust conventions.

Deployment

Follow this to deploy the platform’s services.

Spec

This is the specification describing the deployment of the services. Currently, all services are deployed on a single Digital Ocean Droplet.

TODO: unconsidered: coordinating breaking changes. perhaps a non-issue, provided all changes are backwards compatible - rolling migrations.

Tooling

• Caddy
• Docker
• GitHub
• Google Cloud
• Komodo
• release-please

Caddy

As public services are all hosted on the same VM, Caddy handles routing requests from the various domains to the service ports. The domains are DNS-only, A records registered on Cloudflare, all pointing at the public IP of the VM.

The main deploy/caddy/Caddyfile is the VM entrypoint, interpolating domains and reverse-proxying to container aliases and ports:

{$KOMODO_DOMAIN} {
  respond /ws/periphery 404
  reverse_proxy core:9120
}

{$AUTH_API_DOMAIN} {
  reverse_proxy auth-api:13001
}

{$EXAMINER_DASHBOARD_DOMAIN} {
  reverse_proxy examiner-dashboard:13002
}

{$WEB_APP_DOMAIN} {
  reverse_proxy web-app:13003
}

All public apps need to be added to this config. The interpolated values come from deploy/.env.

Notes

• The Komodo domain proxy should not expose the periphery
  • 404 for /ws/periphery

Docker

All services/components are containerized, and a docker network (platform) is created for the services to communicate through. deploy/bootstrap.sh creates this network on initial creation.

Komodo manages the container stacks.

GitHub

The images are built on GHA runners, and pushed to DOCR.

• docs.yml
  • on pushes to main where docs/ changes, GH Pages deployment is made
• release-container.yml
  • builds then pushes the containers to DOCR, then calls deploy.yml
• release.yml
  • release-please pr action
  • calls release-container.yml

Google Cloud

The Examiner Dashboard and Exam Platform apps use Google OAuth. These two clients need to be created in Google Cloud Console.

Dashboard callback: https://examiner-dashboard.freecodecamp.dev/auth/google/callback. Optional Komodo callback: https://exam-platform.freecodecamp.dev/auth/google/callback. Substitute configured domains; register each URI exactly. See setup for required dashboard variables.

Notes

• Clients are scoped as being internal
• Servers perform check that email matches only @freecodecamp.org addresses

Komodo

Komodo manages the containers. Its own compose file lives in deploy/. It does not manage itself.

Komodo handles rollbacks by going to the previous healthy image, or just with another GitOps deployment. There is a deploy “Action” in Komodo that is called through the deploy GHA calling a webhook with:

• APP - name of app being deployed
• IMAGE - registry URL or previous

release-please

There is one release-please config per service. A change to that service creates a release PR. Upon merge, GHA builds the service image and pushes to DOCR.

A release always builds in the same run that created it. So, no release is orphaned and none is built twice. If a release-please step fails, the other apps’ releases still build (continue-on-error), and the last step fails the job.

Setup

DigitalOcean

Create the VM. A 2vCPU, 4GB RAM, 40GB disk instance is enough to get started.

Create a firewall rule to accept: Incoming SSH:22, HTTP:80, HTTPS:443; Outgoing ICMP, TCP:ALL, UDP:ALL

Ideally, create the VM in the same region as the image registry.

Cloudflare

Create A records for all public domains pointing as DNS-only to the VM IPv4.

SSH and Bootstrap

ssh root@<vm-ip>
git clone https://github.com/freeCodeCamp/exam-platform.git /opt/exam-platform
/opt/exam-platform/deploy/bootstrap.sh

NOTE: bootstrap.sh needs to be re-run any time there is a change to the Caddyfile.

Google Login (Optional)

Komodo reads login providers from Core’s environment only - the Komodo UI and API cannot set them. bootstrap.sh leaves them empty and disabled.

1. Google Cloud Console: create an OAuth client (web application) with redirect URI https://exam-platform.freecodecamp.dev/auth/google/callback.
2. On the Droplet, set in /opt/exam-platform/deploy/.env:

KOMODO_GOOGLE_OAUTH_ENABLED=true
KOMODO_GOOGLE_OAUTH_ID=<client_id>
KOMODO_GOOGLE_OAUTH_SECRET=<client_secret>

3. Restart Core

cd /opt/exam-platform/deploy
docker compose up -d core

NOTE: Re-running bootstrap.sh also applies it.

Admin still needs to create users, because sign-up is closed. Once a user logs in with password, they can link their Google account.

Examiner Dashboard OAuth

Create a separate Google OAuth web client and register https://examiner-dashboard.freecodecamp.dev/auth/google/callback. Login checks for a verified @freecodecamp.org Google Workspace account.

Generate cookie key with openssl rand -hex 64. Create these Komodo Variables before syncing/deploying the dashboard:

Examiner Database

Until production moves to hosted Postgres, eDb runs as Komodo stack exam-platform-examiner-db (databases/examiner/compose.yaml). It holds one schema per app plus shared, all owned by role edb_migrator; the apps connect as roles edd and aapi, which cannot change the schema. Generate four passwords with openssl rand -hex 32, then create secret Variables:

• EXAM_PLATFORM_EXAMINER_DB_PASSWORD: superuser postgres. Bootstrap only.
• EXAM_PLATFORM_EXAMINER_DB_MIGRATOR_PASSWORD, EXAM_PLATFORM_EXAMINER_DB_EDD_PASSWORD, EXAM_PLATFORM_EXAMINER_DB_AAPI_PASSWORD: roles edb_migrator, edd and aapi.
• EXAM_PLATFORM_EXAMINER_DB_MIGRATOR_URL: postgres://edb_migrator:<migrator password>@examiner-db:5432/examiner. Both backend stacks pass it to their migrate service only.
• EXAM_PLATFORM_EXAMINER_DASHBOARD_EDB_DATABASE_URL: postgres://edd:<edd password>@examiner-db:5432/examiner.
• EXAM_PLATFORM_AUTH_API_EDB_DATABASE_URL: postgres://aapi:<aapi password>@examiner-db:5432/examiner.

Deploy the eDb stack by hand before the first aAPI and eDd deploys. The deploy Action does not touch it. initdb/ creates the roles and schemas on an empty volume only: a volume from before edb_migrator existed has app-owned schemas and per-app migration tables, so delete it first. The deploy Action runs the image’s edb-migrate before every backend deploy, so aAPI and eDd deploy in any order. EXAM_PLATFORM_<APP>_MIGRATE_IMAGE Variables are created by the Action.

Komodo Log-In

1. Open https://exam-platform.freecodecamp.dev and log in as admin with the password from .env. Change the password, then add a passkey or TOTP. Nobody can sign up: admins add users under Settings → Users, and each user adds 2FA at first login.
2. Check that server exam-platform reports OK. Its Periphery connects on start.
3. Settings → Providers: add an image registry account named exam-platform for registry.digitalocean.com with a DigitalOcean token scoped to registry read.
4. Settings → Variables: create secret Variables EXAM_PLATFORM_AUTH_API_SENTRY_DSN, EXAM_PLATFORM_EXAMINER_DASHBOARD_SENTRY_DSN.
5. Settings → Users: create service user exam-platform-ci and an API key for it. Store the key in GitHub: Setup.
6. Syncs → new Resource Sync:
   • Repository freeCodeCamp/exam-platform, branch main, resource path deploy/exam-platform.toml.
   • Include user groups on, include variables off, delete off.
   • Execute it twice. The first run creates the Action after the user group, so the group’s Execute permission on it only applies on the second; until then, CI’s RunAction fails with “User does not have required permissions on this Action”. Later edits to the file apply when an admin executes the sync again; it lists pending changes.

The first deploy of each app creates its EXAM_PLATFORM_<APP>_IMAGE Variables. Until eDd’s first deploy, its name answers 502.

GitHub

• Settings → Actions → General: allow GitHub Actions to create and approve pull requests for release-please.
• Environment staging: required reviewers, deployment branch main.
  • Secrets KOMODO_API_KEY and KOMODO_API_SECRET: the API key of Komodo service user exam-platform-ci.
  • Variable KOMODO_URL: https://exam-platform.freecodecamp.dev.
• Repository secret DIGITALOCEAN_USERNAME: DigitalOcean account email used for registry login.
• Repository secret DIGITALOCEAN_ACCESS_TOKEN: DigitalOcean token with registry read/write.
• Repository variable DOCR_REGISTRY: the registry name.
• Branch protection: require CI passed. Release PRs opened with GITHUB_TOKEN trigger no CI, so either allow merging them without the check, or give release-please a GitHub App token.

The secrets and variables, with the GitHub CLI:

gh secret set KOMODO_API_KEY --env staging
gh secret set KOMODO_API_SECRET --env staging
gh variable set KOMODO_URL --env staging --body https://exam-platform.freecodecamp.dev
gh secret set DIGITALOCEAN_USERNAME
gh secret set DIGITALOCEAN_ACCESS_TOKEN
gh variable set DOCR_REGISTRY --body <registry>

Environment Variables

Environment variables are a question mark

Questions

• many .env files, duplication of things like SUCH_SUCH_DOMAIN
• many .env files, but root .env for any common vars
• exam-platform/env/common.env
• separate variables vs secrets (different files)
• variables in .env, secrets in a secret store (cloud)
• build vs runtime

Variables

“Where Set” lists the place that supplies the value. .env is the app’s own .env (copy .env.example). Komodo means the stack environment in deploy/exam-platform.toml.

apps/auth-api

apps/examiner-dashboard

migrate service (apps/auth-api, apps/examiner-dashboard)

One-shot service in each backend’s compose.yaml, profile migrate: runs the image’s edb-migrate. Gets only these.

migrate-cdb service (apps/examiner-dashboard only)

One-shot service in eDd’s compose.yaml, profile migrate: runs the image’s cdb-migrate, after migrate. Gets only these. aAPI never migrates cDb.

Native dashboard execution needs exported environment variables; copying .env alone does not load them. Export them first: set -a; source .env; set +a. Mock mode defaults to test credentials/key and http://127.0.0.1:13002/auth/google/callback. Both need a running, migrated eDb.

apps/web-app

apps/desktop-app

databases/curriculum

deploy/

Set in deploy/.env on the Droplet. bootstrap.sh copies .env.example, then fills each empty secret once with openssl rand -hex 32.

Komodo variables

Per app, EXAM_PLATFORM_<APP>_, with <APP> one of AUTH_API, EXAMINER_DASHBOARD, WEB_APP.

Dashboard, aAPI, eDb and cDb also require these Komodo Variables (no defaults):

GitHub Actions

Housekeeping

Komodo is not in the request path. So, a downed Komodo only affects the automated deploy path. Docker restarts the app containers. Caddy is in the path. So, a Caddyfile change reloads without dropping connections - a new Caddy image restarts it.

Update

• Merge to main
• Run /opt/exam-platform/deploy/bootstrap.sh on the Droplet.

Upgrade

Komodo

• Read the release notes
• Bump the komodo-core and komodo-periphery tags together in compose.yaml through a PR
• MongoDB stays on 8.0: each update pulls its newest patch release
  • A major upgrade waits for Komodo to ship a driver tested against it, then needs MongoDB’s featureCompatibilityVersion steps

Caddy

• bump its tag in a PR

Ubuntu

• it installs security updates by itself
• Reboot when /var/run/reboot-required exists; every container restarts with Docker

Backups

• Droplet backups (--enable-backups) image the whole disk weekly.
• Komodo’s default Backup Core Database Procedure dumps the database daily at 01:00 UTC to /etc/komodo/backups, and keeps the newest 14. The dumps are unencrypted and on the Droplet: copy them elsewhere, encrypted, if the Droplet backups are not enough.
• Restore into an empty database: stop Core and Periphery, drop the database, restore, then start them again.

cd /opt/exam-platform/deploy && set -a && . ./.env && set +a
docker compose stop core periphery
docker compose exec mongo mongosh --quiet -u "$KOMODO_DATABASE_USERNAME" -p "$KOMODO_DATABASE_PASSWORD" \
--authenticationDatabase admin --eval 'db.getSiblingDB("komodo").dropDatabase()'
docker run --rm --network komodo_default -v /etc/komodo/backups:/backups \
-e KOMODO_CLI_DATABASE_TARGET_ADDRESS=mongo:27017 \
-e KOMODO_CLI_DATABASE_TARGET_USERNAME="$KOMODO_DATABASE_USERNAME" \
-e KOMODO_CLI_DATABASE_TARGET_PASSWORD="$KOMODO_DATABASE_PASSWORD" \
-e KOMODO_CLI_DATABASE_TARGET_DB_NAME=komodo \
ghcr.io/moghtech/komodo-cli:2.3.3 km database restore -y
# newest; or --restore-folder <dated folder>
docker compose start core periphery

Disk

• The exam-platform-prune-images Procedure removes unused images every Sunday at 03:00 UTC
• Container logs rotate (Docker’s local driver)
  • Watch /etc/komodo/backups and docker system df

Rotate

• CI key: create a new API key for exam-platform-ci, replace both GitHub secrets, then delete the old key.
• DOCR token: update the registry account under Settings → Providers.
• App secrets: update the secret Variable, then redeploy the stack. Variables apply at deploy.
• Login passwords: in the UI. KOMODO_INIT_ADMIN_PASSWORD only applies to an empty database.
• KOMODO_JWT_SECRET: edit .env, then docker compose up -d core. Everyone is logged out.
• MongoDB password: change it with mongosh (db.changeUserPassword), then edit .env and run docker compose up -d core. The init variables only apply to an empty volume.

Logs

cd /opt/exam-platform/deploy
docker compose logs -f core periphery caddy

Do not:

• Manage this compose project from Komodo.
• Publish MongoDB, or port 9120 beyond 127.0.0.1.
• Remove the /ws/periphery block: no remote servers connect.

Google OAuth

• Set KOMODO_GOOGLE_OAUTH_ENABLED=true, KOMODO_GOOGLE_OAUTH_ID and KOMODO_GOOGLE_OAUTH_SECRET in .env
• docker compose up -d core
  • The Komodo UI and API cannot set login providers
• The Google Cloud OAuth client (web application) needs redirect URI https://<KOMODO_DOMAIN>/auth/google/callback
• Sign-up stays closed, so users link Google to an account an admin created

Releases

Exams are on-demand - there can be no deploy freeze.

Every app that builds is its own release-please component.

Consider release service to perform smart releases - rApp

Apps

Workflow

1. Merges to main cause release.yml release-please PRs for affected applications.
2. Merged release-please PRs call release-container.yml to build the apps, pushes them to the registry, and calls deploy.yml.
3. deploy.yml awaits staging GH environment approval, then tells Komodo to run deploy action. For aAPI and eDd, the Action first runs the image’s edb-migrate, and deploys only if it succeeds.

Rollback

Stacks read compose.yaml from main, so a rollback runs the old image with the current compose file: see Rollback across compose changes.

Rollback reverts the app image only, never eDb’s schema: the old image runs against the newer schema. Migrations keep it backward compatible: drop or rename only once no running or previous image uses the object.

Through GitHub

Go to Actions → Deploy → Run workflow, with the app and previous, or with any digest from a release’s Image line. previous deploys _IMAGE_PREVIOUS, and on success swaps it with _IMAGE_HEALTHY.

Without GitHub

aAPI and eDd: set EXAM_PLATFORM_<APP>_MIGRATE_IMAGE to the digest, run the stack’s migrate service, and continue only if it exits 0. Then set EXAM_PLATFORM_<APP>_IMAGE in Komodo, and deploy the stack. Once it is healthy, set _IMAGE_HEALTHY to the same digest: the next failed deploy rolls back to it. Komodo runs Actions from the UI without arguments, so the Action cannot do this.

Rollback Across Compose Changes

A deploy pairs two inputs from different places: the image digest from the Komodo Variable, and compose.yaml from main at the moment of the deploy. Therefore, a rollback runs the old image with the newest compose file. That works while the compose file changes only in ways the old image tolerates.

• Safe: adding environment variables (an image ignores those it does not read), labels, logging or resource limits; changing the restart policy or a published host port.
• Unsafe: renaming or removing a variable it still reads, or changing what one means; changing LISTEN_ADDR, the container port, WEB_DIR or another path baked into the image; dropping a network alias that another app, or Caddy, still calls.

Keep compose changes backward compatible for one release, as with database migrations: add, release, then remove in a later release. Commit them as fix or feat so they ship with the app’s next release. A chore change sits on main until the app’s next deploy, and that may be a rollback.

To roll back across an incompatible compose change:

1. Preferred: revert the compose change on main, then deploy previous.
2. Emergency: in Komodo, set the stack’s Commit to the commit of the release being restored (GitHub tag <app>-v<version>), then deploy previous. Its compose.yaml matches the image again. Clear the Commit once main is fixed. Until then the sync reports the stack as changed, and running the sync reverts it to main.

Conventions

Coding conventions for this monorepo.

Databases

Structure

Databases in this monorepo are Rust crates consumable by the apps - database interfaces and helpers.

There is one schema per application, and a shared schema for tables that are written to by multiple apps.

This means there is one module per application, and one shared module.

sqlx

sqlx is the database driver. The main API used is the macro API for compile-time checks:

Compile Time

Using the macro API, queries are statically checked during compilation.

#![allow(unused)]
fn main() {
// let pool = <impl sqlx::Executor>;
let user = sqlx::query!("SELECT id, name FROM users WHERE email = $1", email)
    .fetch_one(&pool)
    .await?;

// Typed fields, known at compile time
let name: String = user.name;
}

Macro API

The macros check every query against the database schema at compile time. SQL syntax, table, and column names, and the number and types of parameters are all validated by the build. Output types are inferred from the schema, including nullability: a nullable column becomes an Option, so a missing NULL check cannot compile.

#![allow(unused)]
fn main() {
pub struct User {
    pub id: Uuid,
    pub name: String,
    /// Nullable column, so must be `Option`.
    pub google_id: Option<String>,
}

let user = sqlx::query_as!(
    User,
    "SELECT id, name, google_id FROM users WHERE email = $1",
    email
)
.fetch_optional(&pool)
.await?;
}

No FromRow derive is needed. If a column is renamed, dropped, or changes type in a migration, every query using it fails to build, rather than failing at runtime. Tests are still needed for behaviour, but not to catch typos or schema drift.

The cost is that compilation needs the schema. The macros can be configured to not require a live database connection for compilation, but it requires a couple extra steps:

1. cargo install sqlx-cli
2. Set DATABASE_URL
3. Run database server
4. cargo sqlx prepare
5. git commit .sqlx/

Project can now be built without a database connection (DATABASE_URL must be unset, otherwise it will still try to connect). To update the generated file:

cargo sqlx prepare

To ensure .sqlx/ is kept up-to-date, both with the queries in project and database schema itself, CI runs cargo sqlx prepare --check against a freshly migrated database.

Rust

Structure

For a server with REST API and static file server.

app/
| client/
| dist/     # Build of client
| server/
  | routes/   # Add all routes to their own module here
    | mod.rs  # Export modules from here
  | app.rs    # App router
  | config.rs # Config (mostly env vars) + server state unique to this app
  | main.rs   # In charge of tooling setup + teardown
| Cargo.toml

app.rs is a module that constructs the application router, not a separate library crate. The binary entrypoint main.rs loads configuration, initializes telemetry, and starts the runtime through shared serving helpers.

New routes are added as modules to server/routes/

Misc

To Keep In Mind

• attempts need dApp and wApp version
• backups/snapshots should be taken on cadence, as well as immediately before a deploy
• consider deployment strategy for services
  • in general, forcing backwards compatible bumps is only way forward
  • each service is deployed with redundancy (rollover)
  • clients and servers should have round-one version compatibility
    • forced client updates

Examiner Dashboard

• ability to delete branches
• separate branch management from change requests
  • promoting dev -> stg -> prd should be its own thing