Session: Backend Cleanup & Service Independence

Date: 2026-05-27 Scope: pipeline/backend/refactor/, pipeline/backend/database/migrations/, pipeline/backend/tests/, pipeline/backend/docker-compose.yml, pipeline/backend/Caddyfile, knowledge/, .opencode/skills/

Summary

Removed three orphaned directories from pipeline/backend/: the refactor/ migration project, the tests/ integration suite, and the central docker-compose.yml orchestrator. Migrated database schema knowledge to knowledge/. Each backend service now runs independently via its own docker-compose.yml on a shared dev network.

Key Learnings

New Patterns

• Migrations belong as numbered directories under the service they target (database/migrations/001_initial_migration/), not as standalone projects. They are one-shot scripts tracked in git but not installed as packages. No __init__.py, no pyproject.toml — just standalone .py files with relative imports.
• One-shot scripts should not become permanent packages. The refactor project had its own pyproject.toml, uv.lock, .venv/ — overhead for scripts that ran once and are done. If there’s a chance of reuse, the pattern is: loose scripts in a migrations/ or scripts/ dir, using the project’s installed dev dependencies.
• Docker Compose include: is convenient but couples services. Removing the central orchestrator and running each docker-compose.yml independently works because all sub-composes define networks: dev: name: dev — Docker reuses the named network. No config changes needed.
• Named Docker volumes survive docker compose down. Only docker compose down -v or explicit docker volume rm removes them. This makes the include: → independent transition safe for stateful services like Infisical’s PostgreSQL.

Decisions

• Delete migration scripts rather than preserve them in-tree. The v0→v1 migration is completed. The scripts are single-use and the knowledge (what was transformed, results) is preserved in knowledge/journal/fixes/db-v0-to-v1-migration.md. Git history has the code if ever needed.
• Delete pipeline/backend/tests/. SDK integration tests overlap with lemna-sdk’s own tests. Endpoint integration tests exist in each endpoint repo. The standalone test project added maintenance burden for rarely-run tests.
• Delete central docker-compose.yml and Caddyfile. Each service manages its own compose. Caddyfile was entirely commented out — no active routes.
• Historical journal sessions are not rewritten. References to deleted paths (pipeline/backend/refactor/, pipeline/backend/tests/) in knowledge/journal/sessions/done/ are left as-is — they accurately describe what happened at the time. Only active/planning docs and skills are updated.

Pitfalls

• Docker Compose warns about pre-existing volumes and networks when running sub-composes independently after a central compose is removed. These are benign — volume "infisical-pg_data" already exists but was not created by Docker Compose just means compose didn’t create it this run. Adding external: true to the volume/network definitions would suppress the warnings but isn’t required.
• The include: compose pattern creates project-scoped networks (e.g., backend_infisical) that differ from standalone project networks (e.g., secrets_infisical). After transitioning, the containers join the secrets_infisical network instead. This doesn’t affect functionality since inter-service communication uses the shared dev network, not the project-scoped one.
• Git submodules look like directories but have their own .git. The refactor/ and tests/ directories were NOT submodules (their .git pointed to the parent repo), which made them safe to delete without submodule commands.

Skill Updates Needed

• map skill: pipeline/backend/ entry needs updating — remove tests/ reference, note that services run independently (no central compose)
• essentials skill: Add note about named Docker volumes surviving docker compose down (not a pitfall, but worth knowing)
• patterns skill: Add the migrations/001_X/ pattern for one-shot database migration scripts

Files Modified

• Created: knowledge/journal/fixes/db-v0-to-v1-migration.md, knowledge/journal/fixes/db-v0-to-v1-exploration-report.json
• Created then deleted: pipeline/backend/database/migrations/001_initial_migration/ (created, then deleted by user decision)
• Deleted: pipeline/backend/refactor/ (entire directory), pipeline/backend/tests/ (entire directory), pipeline/backend/docker-compose.yml, pipeline/backend/Caddyfile
• Updated: pipeline/backend/database/.gitignore (reverted — removed !*.json and !*.sh), pipeline/backend/database/pyproject.toml (reverted — removed migration dev deps)
• Updated: .opencode/skills/secrets/SKILL.md (removed tests/conftest.py callout, updated .env references)
• Updated: .opencode/skills/map/SKILL.md (removed migrations/ entry, needs further update for tests/ and central compose)
• Updated: knowledge/index.md (added db-v0-to-v1 migration entry under Fixes)
• Updated: knowledge/plans/local-dev-test-plan.md (noted backend tests removed, updated commands)
• Updated: knowledge/plans/monorepo-decomposition-plan.md (resolved tests question as deleted, updated migration steps)