Lemna Knowledge Base

4 min read

title: Local Dev Environment Orchestration tags: [docker, dev-environment, infrastructure, session] created: 2026-05-26 updated: 2026-05-26 status: active related:

Session: Local Dev Environment Orchestration

Date: 2026-05-26 Scope: pipeline/backend/docker-compose.yml, pipeline/backend/Caddyfile, pipeline/backend/{secrets,bucket,database}/docker-compose.yml, .gitignore

Summary

Created a unified Docker Compose orchestration at pipeline/backend/ using the include: directive, a shared dev network, and fixed-name volumes. Moved Caddy from the database stack to the orchestration level as a routing layer. Recovered Infisical and RustFS data from orphaned volumes caused by project name changes.

Key Learnings

New Patterns

  • Docker Compose include: directive merges sub-compose files into one project without duplicating definitions. Paths in include: resolve relative to the file containing them.
  • Shared dev network with name: dev gives a fixed Docker network name independent of the compose project name. Services across all included files join dev for cross-stack communication while keeping their own networks for internal isolation.
  • Fixed-name volumes with name: prevent data loss when compose project name changes. Without name:, Docker prefixes volumes with the project name (secrets_pg_data vs backend_pg_data), creating empty new volumes and orphaning old ones.
  • Caddy belongs at orchestration level, not inside a specific service stack. It routes across all services (API, secrets dashboard, storage console, Mongo admin).
  • INFISICAL_API_URL must differ host vs container: host uses http://localhost:8192, containers use http://infisical-backend:8080 (container name + internal port). The database compose hardcodes the container URL.

Decisions

  • No init scripts or bucket-init services. One-time setup (creating S3 buckets, seeding Infisical) is manual. Keeps compose minimal.
  • No justfile targets for dev-up/dev-down. docker compose up -d from pipeline/backend/ is sufficient.
  • Caddyfile with commented-out subdomain routing. User prefers subdomain-based routing; path-based routes commented out as placeholder until subdomain config is ready.
  • Mongo Express port 8081 published directly. Previously only accessible via Caddy proxy; now accessible directly for dev.
  • Orchestrating compose lives at pipeline/backend/, not monorepo root. It only orchestrates backend services — placing it at root is misleading.

Pitfalls

  • Volume data loss on project name change. Running docker compose from a different directory changes the project name, which creates new empty volumes. Fix: use name: on all volumes. Recovery: docker run --rm -v old_vol:/source -v new_named_vol:/target alpine cp -a /source/. /target/.
  • Docker Compose warns “volume already exists but was not created by Docker Compose” when a name: volume was created externally (e.g., by cp migration). Warnings are harmless and disappear if volumes are recreated by Compose.
  • Cannot mix list-style and map-style env vars in same environment: block. - INFISICAL_API_URL (passthrough) and INFISICAL_API_URL: http://... (map) in the same block causes YAML parse errors. Convert entire block to one style.
  • include: cannot override services from included files. To modify a service (e.g., add a network), you must edit the source compose file directly.
  • Mongo Express ME_CONFIG_SITE_BASEURL=/mongo only makes sense when proxied via Caddy at /mongo*. For direct access, remove it or Mongo Express fails to serve at root path.

Skill Updates Needed

  • map skill: Add pipeline/backend/docker-compose.yml and pipeline/backend/Caddyfile to the backend directory structure. Note the orchestrating compose with include:.
  • essentials skill: Add Docker Compose gotchas: (1) volume project-name prefix causes data loss — use name: on all volumes; (2) include: cannot override included services; (3) mixing env var syntax styles in same block fails.
  • secrets skill: Note that INFISICAL_API_URL is http://localhost:8192 for host but http://infisical-backend:8080 for containers (hardcoded in database compose). Update the “Container can’t reach Infisical” gotcha.
  • map skill: Update port table to include all 5 direct HTTP ports: 8000 (Database API), 8192 (Infisical), 9000 (RustFS S3), 9001 (RustFS Console), 8081 (Mongo Express).

Files Modified

  • pipeline/backend/docker-compose.yml — new: orchestrating compose with include:, dev network, Caddy service, fixed-name caddy volumes
  • pipeline/backend/Caddyfile — new: moved from database/ with commented-out subdomain routing placeholder
  • pipeline/backend/secrets/docker-compose.yml — added dev network to backend, fixed-name volumes (infisical-pg_data, infisical-redis_data)
  • pipeline/backend/bucket/docker-compose.yml — added dev and default networks, fixed-name volumes (rustfs-data, rustfs-logs)
  • pipeline/backend/database/docker-compose.yml — added dev network to app and mongo-express, hardcoded INFISICAL_API_URL, removed Caddy service, added mongo-express port 8081, removed ME_CONFIG_SITE_BASEURL
  • pipeline/backend/database/Caddyfile — deleted (moved to pipeline/backend/Caddyfile)
  • .gitignore — added !docker-compose.yml

Graph View

Recent Notes

Backlinks