Skip to content

Publish dev database Docker images to GHCR#837

Merged
marcodejongh merged 10 commits intomainfrom
publish_dev_db
Feb 14, 2026
Merged

Publish dev database Docker images to GHCR#837
marcodejongh merged 10 commits intomainfrom
publish_dev_db

Conversation

@marcodejongh
Copy link
Owner

@marcodejongh marcodejongh commented Feb 14, 2026

Summary

  • Pre-populated dev database image: New Dockerfile.dev-db builds a PostgreSQL + PostGIS image with all Kilter/Tension board data and drizzle migrations pre-applied. Developers pull and run — no more waiting for ~400MB APK downloads and pgloader imports on first setup.
  • Simplified db:up: Moved from a fragile one-liner to scripts/dev-db-up.sh shell script. It starts containers, runs migrations (to pick up any newer than the image), and downloads/imports MoonBoard data.
  • CI workflow: Added build-dev-db job to publish the image to GHCR on merge to main.

Breaking changes

Volume path changed

The postgres data directory moved from /var/lib/postgresql/data to /var/lib/postgresql/pgdata (to work around Docker's VOLUME declaration that discards build-time data). Existing db_data volumes won't be picked up by the new image. To reset:

docker compose down -v   # removes the old volume
docker compose up -d     # pulls new image, creates fresh volume
npm run db:up            # runs migrations + MoonBoard import

db:setup script removed

The npm run db:setup command and the db_setup Docker service have been removed. All board data (Kilter/Tension) is now baked into the postgres image. MoonBoard data is imported by npm run db:up.

MoonBoard data not in the image

MoonBoard data requires the Neon HTTP proxy (TypeScript import script), so it can't be imported during Docker build. It's downloaded and imported by npm run db:up on first run instead.

Technical details

  • PGDATA trick: ENV PGDATA=/var/lib/postgresql/pgdata avoids the official postgres:17 image's VOLUME /var/lib/postgresql/data which would discard build-time data
  • pgloader via TCP: pgloader can't parse %2F-encoded Unix socket URLs, so postgres listens on localhost TCP for pgloader while psql uses Unix sockets
  • Drizzle migrations in Docker: Replicates drizzle-orm's migration runner using jq + psql + sha256sum to parse the journal, apply SQL files, and record hashes
  • Multi-platform: Builds for both linux/amd64 and linux/arm64

Test plan

  • Verify build-dev-db workflow builds successfully on this PR
  • After merge, trigger workflow_dispatch and verify images appear in GHCR
  • Run docker compose down -v && docker compose up -d && npm run db:up locally
  • Verify tables exist: psql -h localhost -U postgres main -c "\dt kilter_*"
  • Verify app works end-to-end with the new setup

🤖 Generated with Claude Code

Add a GitHub Actions workflow to build and publish the postgres-postgis
and db-setup Docker images to GHCR, enabling CI to use the same database
environment as local development. Update CI workflows and docker-compose
to reference the published images.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Feb 14, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
boardsesh Building Building Preview, Comment Feb 14, 2026 8:50am

Request Review

@chatgpt-codex-connector
Copy link

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - No significant issues found.

The workflow is well-structured with appropriate permissions, multi-platform builds, and proper conditional push logic for PRs vs main branch.

Build a multi-stage Dockerfile (Dockerfile.dev-db) that downloads Kilter/Tension
APKs, extracts SQLite databases, fixes compatibility issues, and imports all data
via pgloader at build time. This eliminates the ~400MB download + import step that
previously ran on every developer's first `db:up`.

- Stage 1: postgres:17 + PostGIS (self-contained, no GHCR dependency)
- Stage 2: Downloads APKs, extracts/fixes SQLite DBs, imports via pgloader
- Stage 3: Clean image with only the pre-populated PGDATA
- Uses PGDATA=/var/lib/postgresql/pgdata to avoid Docker VOLUME discard
- docker-compose.yml uses new dev-db image, removes db_setup service
- GitHub Actions workflow builds/publishes ghcr.io/marcodejongh/boardsesh-dev-db

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. package.json:27 - db:setup script broken: The PR removes db_setup service from docker-compose.yml, but package.json still references it via npm run db:setup which runs docker-compose up db_setup. This will fail with "no such service: db_setup".

  2. package.json:26 - db:up check no longer makes sense: The db:up script checks for packages/db/docker/tmp/db-setup-complete.flag which was created by the now-removed db_setup service. With pre-built images that include data, this flag won't exist and users will get an error telling them to run the broken db:setup command.

  3. MoonBoard data missing from dev-db image: The original setup-development-db.sh downloads MoonBoard problem data (lines 80-94), but Dockerfile.dev-db doesn't include this. After merge, npm run db:up will fail at the npm run db:import-moonboard step unless MoonBoard data is handled separately.

  4. Dockerfile.dev-db:44 - Potential nested APK failure: For Tension APK, if nested extraction fails (unzip -o -j inside the || clause), the build continues but mv will fail on missing db.sqlite3. Consider adding error handling for this edge case.

Notes

  • The README/docs updates for the new workflow (npm run db:setup → just docker-compose up) should be added, but can be done in a follow-up.
  • Consider updating package.json scripts in this PR to match the new Docker-based workflow.

- Remove db:setup script (referenced removed db_setup service)
- Fix db:up: remove stale db-setup-complete.flag check, auto-download
  MoonBoard data if not already present
- Add explicit verification after Tension APK extraction to fail fast
  with a clear error instead of silently continuing
- Apply drizzle migrations inside Docker build via psql + journal.json,
  recording SHA-256 hashes in __drizzle_migrations table so drizzle-orm
  recognizes them as already applied
- Widen build context to packages/db/ so Dockerfile can access both
  docker/ scripts and drizzle/ migrations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. Data volume incompatibility (docker-compose.yml): Volume mount changes from db_data:/var/lib/postgresql/data to db_data:/var/lib/postgresql/pgdata. Existing developers with populated db_data volumes will lose their data or get startup errors. Consider documenting a migration step (e.g., docker volume rm boardsesh_db_data) in the PR description.

  2. External download dependency in image build (Dockerfile.dev-db:62-70): The build downloads APKs from d.apkpure.net at build time. If this third-party site is unavailable or rate-limits GitHub Actions, the CI build will fail. This is acceptable for dev tooling but worth noting for reliability.

  3. Missing db_setup service removal documentation: The db_setup service was removed from docker-compose.yml, but the PR description doesn't mention that npm run db:setup is removed from package.json. The old workflow (npm run db:setup then npm run db:up) is replaced with just npm run db:up, which is good but should be documented for existing contributors.

  4. Hardcoded database name mismatch: Dockerfile.dev-db:109 creates database named "main", while the old setup-development-db.sh:18 defaulted to "verceldb". The docker-compose.yml sets POSTGRES_DB=main, so this is consistent, but any developers with old .env files pointing to "verceldb" will need to update.

Minor

  • package.json:26: The inline script downloads MoonBoard data synchronously on every db:up check. The conditional guards it, but it adds complexity to what was a simple command. Consider extracting to a script file for maintainability.

Without this flag, psql silently continues past SQL errors and returns
exit code 0, meaning broken migrations could be applied without failing
the build. Also adds progress counter (e.g. [3/56]) and a final success
message to confirm all migrations completed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. packages/db/docker/Dockerfile.dev-db:54 - Using unquoted $DB_FILE in script and shell commands. If path contains spaces, it will break. Use "$DB_FILE" consistently.

  2. packages/db/docker/Dockerfile.dev-db:53-65 - APK download URLs from d.apkpure.net are third-party and could change or become unavailable. Consider documenting a fallback or pinning to specific versions.

  3. .github/workflows/dev-db-docker.yml - The build-db-setup job (lines 126-187) builds an image from ./packages/db/docker/Dockerfile, but the docker-compose.yml changes remove the db_setup service entirely. This orphaned image may never be used.

  4. docker-compose.yml - Volume path changed from db_data:/var/lib/postgresql/data to db_data:/var/lib/postgresql/pgdata. Existing developers will have data in the old volume mount that won't be visible with the new image. Consider noting this breaking change for existing dev setups.

  5. package.json - The new db:up script uses a long inline curl/unzip chain. If the moonboard zip download fails or is already extracted but corrupted, the script won't detect it. The original two-command approach (db:setup + db:up) had clearer separation of concerns.

Documentation

No documentation files appear to need updates for this change.

- Increase max_wal_size to 2GB to suppress "checkpoints occurring too
  frequently" warnings during heavy import/migration writes
- Add verification step after migrations: queries __drizzle_migrations
  count and compares to expected, fails build on mismatch
- Prints table count for quick visual confirmation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. packages/db/docker/Dockerfile.dev-db:55-57 - APK download URLs use a third-party domain (d.apkpure.net) that could change/break without notice. Consider pinning to specific APK versions or documenting the fallback strategy.

  2. packages/db/docker/Dockerfile.dev-db:77-78 - Nested APK extraction fallback only handles one nesting level. If the APK structure changes further, extraction will silently fail at the subsequent test -f check.

  3. .github/workflows/backend-tests.yml:24 / e2e-tests.yml:26 - Using :latest tag for CI images means tests could break if a bad image is pushed. Consider using SHA-pinned or semver tags for CI stability.

  4. package.json (db:up script) - The inline shell command downloads and unzips files without verifying checksums. Minor reliability concern for developer setup.

Notes

  • No documentation updates needed (no existing docs cover Docker/CI infrastructure).
  • No tests required for infrastructure changes like this.

PostgreSQL TCP socket binding fails under QEMU emulation in Docker
buildx arm64 builds ("could not bind IPv4 address 127.0.0.1: Address
already in use"). Switch to Unix sockets exclusively:

- listen_addresses='' (disable TCP entirely)
- unix_socket_directories=/var/run/postgresql
- All psql calls use -h /var/run/postgresql
- pgloader DB_URL uses URL-encoded socket path (%2Fvar%2Frun%2Fpostgresql)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. Data volume path mismatch may cause data loss on existing setups - docker-compose.yml:12: Changed volume mount from db_data:/var/lib/postgresql/data to db_data:/var/lib/postgresql/pgdata. Existing db_data volumes from previous setups won't be picked up. Consider documenting this in the PR body or README.

  2. db:up script is complex and fragile - package.json:26: The one-liner with nested conditionals, subshells, curl, and unzip is hard to maintain. If curl or unzip fail silently (due to || short-circuit), subsequent commands may run on incomplete data. Consider moving to a shell script.

  3. APK download URLs may be unreliable - Dockerfile.dev-db:57-62: Using apkpure.net URLs for APK downloads during Docker build. These third-party URLs could change or rate-limit, causing build failures. No checksum verification is performed on downloaded APKs.

  4. Missing MoonBoard data in pre-built image - The new Dockerfile.dev-db includes Kilter/Tension data but not MoonBoard. The db:up script downloads MoonBoard separately. This is fine but worth noting the pre-built image is not fully complete without running db:up.

  5. Removed db:setup script without migration path - package.json: The db:setup script was removed. Users with existing workflows relying on this will get an error. Consider documenting the migration path.

Notes

  • No documentation updates needed - docs don't cover Docker/database setup
  • No tests added, but this is infrastructure/CI change - manual verification via the test plan is appropriate

- pgloader can't parse %2F-encoded Unix socket URLs (DIGIT-CHAR-P error).
  Switch to localhost TCP for pgloader while keeping Unix sockets for psql.
- Move complex db:up one-liner to scripts/dev-db-up.sh for maintainability.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

⚠️ Needs attention - Chicken-and-egg problem with GHCR image dependency; CI tests will fail until image is published.

Issues

  1. CI dependency on unpublished image (.github/workflows/backend-tests.yml:24, .github/workflows/e2e-tests.yml:26): The backend-tests and e2e-tests workflows now reference ghcr.io/marcodejongh/boardsesh-postgres-postgis:latest, but this image won't exist until after the PR is merged and the dev-db-docker.yml workflow runs. These CI jobs will fail on this PR and any subsequent PRs until the image is manually published via workflow_dispatch.

  2. Race condition in dev-db-up.sh (scripts/dev-db-up.sh:27): sleep 3 is fragile. If postgres takes longer to initialize, migrations will fail. Consider polling pg_isready similar to the approach used in the Dockerfile.

  3. MoonBoard import runs every time (scripts/dev-db-up.sh:42-43): The check at line 33 only guards the download, but npm run db:import-moonboard runs unconditionally on every npm run db:up. This may cause duplicate import attempts or unnecessary work.

  4. Workflow builds orphaned db-setup image (.github/workflows/dev-db-docker.yml:133-188): The build-db-setup job references ./packages/db/docker/Dockerfile but the PR description says db_setup service was removed. This job may be leftover and could build an unused image.

Port 5432 is claimed under QEMU emulation (buildx arm64 on amd64 runner),
causing "Address already in use" for both IPv4 and IPv6. Using a non-standard
port (15432) avoids the conflict while keeping TCP for pgloader and Unix
sockets for psql.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. scripts/dev-db-up.sh:42 - MoonBoard import runs on every npm run db:up invocation even if data already exists. The check at line 33 only skips download, but line 42-43 runs the import unconditionally. This may cause unnecessary work or duplicate data imports on subsequent runs.

  2. scripts/dev-db-up.sh:27 - Hard-coded sleep 3 for postgres readiness is fragile. Consider using a wait loop with pg_isready like the Dockerfile does, since container startup time can vary.

  3. .github/workflows/dev-db-docker.yml:68 - The build-dev-db job references Dockerfile.dev-db but depends on the boardsesh-postgres-postgis image being published first. However, there's no needs: build-postgres-postgis dependency, so on first run (or if postgres-postgis image doesn't exist in GHCR), this job may fail on PR builds that don't push to registry.

  4. .github/workflows/backend-tests.yml:24 - Changed to use ghcr.io/marcodejongh/boardsesh-postgres-postgis:latest but this image won't exist until after this PR is merged and the workflow runs. This could break CI for the PR itself. Consider keeping the original postgres:16-alpine image until the image is published, or ensure the image build runs before tests.

QEMU emulation on amd64 runners cannot bind any TCP sockets, which
pgloader requires for its connection. Build amd64 only in CI. Apple
Silicon users can pull the amd64 image (Docker Desktop runs via Rosetta)
or build locally where native arm64 has no QEMU limitation.

Also reverts port 15432 workaround since arm64 is no longer built in CI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@claude
Copy link

claude bot commented Feb 14, 2026

Claude Review

Ready to merge - Minor issues noted below, but nothing blocking.

Issues

  1. scripts/dev-db-up.sh:28 - sleep 3 is a fragile wait mechanism. Consider using a health check loop with docker compose exec postgres pg_isready instead.

  2. scripts/dev-db-up.sh:17-38 - MoonBoard import runs on every db:up invocation even when data already exists. The directory check only gates the download, not the import. Consider checking if MoonBoard tables have data before re-importing.

  3. .github/workflows/dev-db-docker.yml:122-145 - The build-db-setup job references ./packages/db/docker/Dockerfile but this seems to be the old setup image. Consider whether this job is still needed given db_setup service was removed from docker-compose.

  4. .github/workflows/backend-tests.yml and e2e-tests.yml - Changed from postgres:16-alpine to ghcr.io/marcodejongh/boardsesh-postgres-postgis:latest. This creates a chicken-and-egg issue on first merge: CI will fail until the image is published. However, since workflows only push on non-PR events, and the image workflow runs on the same triggers, this should resolve itself post-merge.

Documentation

CLAUDE.md was appropriately updated with the new database setup process.

Explain the faster dev setup with the pre-built Docker image, how to
pull/reset/build-locally, and that MoonBoard data is imported on first run.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@marcodejongh marcodejongh merged commit 3da78e8 into main Feb 14, 2026
5 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant