title: Local Dev Setup — Full Test Plan tags: [plan, testing, infrastructure, local-dev, infisical, rustfs, mongodb, pantry] created: 2026-05-26 updated: 2026-05-27 status: active related:

Local Dev Setup — Full Test Plan

Scope: End-to-end verification of Infisical, RustFS, MongoDB, Database API, SDK, Pantry, and Endpoints running locally.

Current State

Infisical: running at localhost:8192
RustFS: running at localhost:9000 (S3) / localhost:9001 (console)
MongoDB: running (via Docker compose)
Database API (backend-app-1): crash-looping — Docker image has old SDK missing load_env. Must rebuild before any integration test works.
Caddy: running on ports 80/443
Mongo Express: running at localhost:8081

Tier 0: Fix the Blocker

Rebuild Database API Docker image
```
cd pipeline/backend/database && just docker-build
```
The current image imports load_env from sdk.environ, but the baked-in SDK only has load_dotenv. Without this fix, no endpoint or integration test can reach the database.

Tier 1: Infrastructure Layer (no GPU needed)

These validate that the three core services are reachable and correctly configured.

1.1 Test Infisical Connectivity

Run a Python snippet that calls load_env() from the SDK and prints all 9 secrets:

from sdk.environ import load_env
load_env()
import os
for key in ["API_URL", "API_KEY", "R2_ENDPOINT_URL", "R2_ACCESS_KEY_ID", "R2_SECRET_ACCESS_KEY", "RUNPOD_API_KEY", "OLLAMA_API_KEY", "LINEAR_API_KEY", "GITHUB_TOKEN"]:
    val = os.environ.get(key, "<MISSING>")
    print(f"  {key}={val[:20]}..." if len(val) > 20 else f"  {key}={val}")

Pass criteria: All 9 secrets resolve. No <MISSING> values.

Key risk: R2_ENDPOINT_URL might point at Cloudflare R2 instead of localhost:9000. Verify/update in Infisical dashboard at http://localhost:8192 → project lemna-platform → environment dev.

1.2 Test RustFS S3 Connectivity

import boto3
from sdk.environ import load_env
load_env()
s3 = boto3.client("s3",
    endpoint_url=os.environ["R2_ENDPOINT_URL"],
    aws_access_key_id=os.environ["R2_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["R2_SECRET_ACCESS_KEY"],
)
print(s3.list_buckets())
s3.put_object(Bucket="lemna-data", Key="test/verify", Body=b"hello")
body = s3.get_object(Bucket="lemna-data", Key="test/verify")["Body"].read()
assert body == b"hello"
s3.delete_object(Bucket="lemna-data", Key="test/verify")
print("RustFS round-trip OK")

Pass criteria: List buckets succeeds, put/get/delete round-trip works. Bucket lemna-data exists (create via aws s3 mb or RustFS console if not).

1.3 Test MongoDB Connectivity

from pymongo import MongoClient
client = MongoClient("mongodb://localhost:27017", serverSelectionTimeoutMs=5000)
client.server_info()
db = client["lemna"]
db["_test"].insert_one({"ok": True})
assert db["_test"].find_one({"ok": True}) is not None
db["_test"].drop()
print("MongoDB round-trip OK")

Pass criteria: Insert, find, drop succeeds.

1.4 Test Database API

After rebuilding the image and restarting:

# Restart with fresh image
cd pipeline/backend && docker compose up -d app
# Wait for it
curl -s http://localhost:8000/docs | head -5

Then test CRUD with X-API-KEY header (value from Infisical API_KEY secret):

POST /jobs — create a job
PATCH /jobs/{id} — update status
POST /jobs/query — query by project/type/status
POST /jobs/get — get by ID list
DELETE /jobs/{id} — delete

Pass criteria: All routes return expected status codes. API key auth works (reject without header, accept with header).

Tier 2: SDK Integration Tests (no GPU needed)

Note: The backend integration test suite (pipeline/backend/tests/) has been removed. SDK tests should be run from pipeline/backend/lemna-sdk/tests/. The tests below are preserved as documentation of what was covered.

Tests formerly covered:

Test	What it exercises
`test_submit_and_update`	Job lifecycle: submit → RUNNING → COMPLETED
`test_query`	Query by project_name/job_name
`test_get_by_id`	Retrieve by ID
`test_wait_polls_until_completion`	Polling wait mechanism
`test_failed_job`	FAILED status + log field
`test_submit_with_binary_file`	CAS dehydrate/hydrate via S3
`test_delete_job`	Delete + CAS purge
`test_cache_hit_returns_same_id`	Input hash caching
`test_no_cache_creates_new_job`	Cache bypass
`test_failed_job_recovery`	Retry after failure
`test_update_nonexistent_job`	404 error handling
`test_status_transition_flow`	QUEUED → RUNNING → COMPLETED
`test_concurrent_submissions`	Thread-safety
`test_query_by_status`	Status-based filtering

Pass criteria: All 11 pass. This exercises SDK client + Database API + S3 CAS end-to-end.

2.2 Manual CAS Round-Trip

from sdk.cas import ContentStore
from sdk.dtypes import BinaryFile
 
cas = ContentStore("tests")
key = cas.put(b"test payload")
data = cas.get(key)
assert data == b"test payload"
 
bf = BinaryFile(path="test.bin", content=b"hello cas")
cas.dehydrate(bf)
assert bf.content is None
assert bf.storage_key is not None
cas.hydrate(bf)
assert bf.content == b"hello cas"
print("CAS round-trip OK")

Pass criteria: Put/get works. Dehydrate clears content and sets storage_key. Hydrate restores content from S3.

Tier 3: Pantry Tests (no GPU needed)

3.1 Test `pantry add` / `pantry status`

cd pipeline/services/endpoint-pesto
pantry status    # show current entries
pantry add .     # add all untracked matching .datapattern
pantry status    # verify new entries appear

Pass criteria: .datapattern matching works. New entries show as added.

3.2 Test `pantry push`

pantry push --dry-run    # preview
pantry push              # upload to RustFS
pantry status             # entries should show as `synced`

Verify objects in RustFS:

aws s3api list-objects-v2 --bucket lemna-data --endpoint-url http://localhost:9000 --no-sign-request
# or check at http://localhost:9001 (RustFS console)

Pass criteria: Files uploaded. Manifest updated with storage_key. Status shows synced.

3.3 Test `pantry pull`

# Delete a pushed file from disk
rm data/some_model.pt    # (pick a safe test file)
pantry pull data/some_model.pt
# Verify file restored and hash matches
pantry verify

Pass criteria: File downloaded from RustFS. SHA-256 hash matches manifest. Verify shows no mismatches.

3.4 Test `pantry resolve`

from pantry.core import resolve
from pantry.repo import find_root
 
root = find_root(Path("pipeline/services/endpoint-pesto"))
path = resolve("data/some_model.pt", repo_root=root)
assert path.exists()

Pass criteria: Returns local path when file exists. Raises DataNotAvailableError when file is deleted from disk (needs pull). Works after pull.

3.5 Test `pantry extern`

# From a consumer project
pantry add data/shared_file.pt --extern pesto-models
pantry resolve data/shared_file.pt    # should follow extern to pesto-models namespace

Pass criteria: Cross-namespace resolution works. Extern entries show e in pantry tree.

3.6 Test `pantry gc`

# After pushing and removing some manifest entries
pantry gc            # list orphaned objects
pantry gc --prune    # delete orphaned objects from S3

Pass criteria: Orphaned objects detected. Prune removes them from S3.

Tier 4: Endpoint Tests (requires GPU)

4.1 Build All Endpoint Images

scripts/apply-all.sh pipeline/services docker-build

4.2 Run Endpoint Integration Tests

# Endpoint tests are now run from each endpoint's own tests/ directory
# e.g. pipeline/services/endpoint-pesto/tests/
HAS_GPU=1 uv run pytest test_endpoint.py -v

Covers 7 endpoints: carbonara, pesto, boltz, boltzgen, carbonara-binders, cpmp, openmm.

Each test: builds job from tests/test_input.json → uploads binary files to CAS → submits via Docker container → waits for COMPLETED status.

Pass criteria: Each endpoint’s Docker container starts, loads model, processes input, returns COMPLETED. Binary output files hydrated successfully from S3.

Single endpoint test:

# From any endpoint directory:
HAS_GPU=1 uv run pytest tests/test_endpoint.py -v -k pesto

Tier 5: Manual End-to-End Workflow

5.1 Full Job Lifecycle via SDK

from sdk import LemnaJobClient, JobBase, JobStatus
from sdk.dtypes import BinaryFile
from sdk.backends import LocalBackend
 
client = LemnaJobClient()
job = JobBase(
    project_name="local-dev-test",
    job_name="smoke-test",
    job_type="test",
    pid=[],
    input_data={"file": BinaryFile(path="input.pdb", content=b"FAKE_PDB_DATA")},
)
backend = LocalBackend()
job_id = client.submit(job, backend=backend)
print(f"Submitted: {job_id}")
 
# Check in Mongo Express at http://localhost:8081 → lemna → Job collection
job.status = JobStatus.COMPLETED
client.update(job)
results = list(client.get([job_id]))
assert results[0].status == JobStatus.COMPLETED
print("E2E lifecycle OK")
client.delete(results[0])

5.2 Batch Pantry Push

scripts/pty-push-all.sh

Pass criteria: No failures across all namespaces.

Execution Order

0 → 1.1 → 1.2 → 1.3 → 1.4 → 2.1 → 2.2 → 3.1 → 3.2 → 3.3 → 3.4 → 3.5 → 3.6

Skip Tier 4 unless GPU is available and images are built. Tier 5 is optional manual smoke testing.

Known Risks

Risk	Mitigation
`R2_ENDPOINT_URL` in Infisical `dev` points to Cloudflare R2, not local RustFS	Update via Infisical dashboard before testing
RustFS bucket `lemna-data` doesn’t exist	Create via `aws s3 mb s3://lemna-data` or console
Database API image stale (current blocker)	Rebuild with `just docker-build`
`R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` don’t match RustFS creds (`lemna-admin` / `lemna-admin-secret`)	Update in Infisical dashboard
Endpoint images not built	Run `scripts/apply-all.sh pipeline/services docker-build`
Test infrastructure conflicts with main compose on port 9000	Stop main compose first, or modify test ports

Explorer