title: Pomodoro v0.8.0 — Low Validation lDDT Root Cause Analysis tags: [journal, pomodoro, edm, bug, sampling, lddt, v0.8] created: 2026-05-11 updated: 2026-05-11 status: active related:
Pomodoro v0.8.0 — Low Validation lDDT Root Cause Analysis
Problem
Validation lDDT during EDM sampling was ~35, while the debug workspace reference achieved ~60. Training-step lDDT (single-step denoising) was unaffected.
Root Cause
sample_schedule had reversed interpolation direction — ascending (sigma_min → sigma_max) instead of descending (sigma_max → sigma_min). The sampler started from near-zero noise and progressively added noise, then jumped to 0.0 at the final padded step. This inverted the entire denoising trajectory.
Buggy code (all versions v0.6.0–v0.8.0):
# Ascending — WRONG
sigmas = (
cfg.sigma_min**inv_rho + steps / (num_steps - 1) * (cfg.sigma_max**inv_rho - cfg.sigma_min**inv_rho)
) ** cfg.rhoCorrected code (matching debug reference):
# Descending — CORRECT
sigmas = (
cfg.sigma_max**inv_rho + steps / (num_steps - 1) * (cfg.sigma_min**inv_rho - cfg.sigma_max**inv_rho)
) ** cfg.rho
sigmas = sigmas * cfg.sigma_dataSecondary Issue: Missing sigma_data Scaling
The debug reference (workspace/debug/edm/debug_training.py) multiplies the schedule by cfg.sigma_data (line 46). All production versions were missing this scaling. Added alongside the direction fix.
Why Training lDDT Was Unaffected
sample_schedule is only called by edm_sample(), which is only invoked in validate(). Training step() uses noise_distribution() for single-step denoising — no schedule involved.
Other Differences Reviewed (Not the Cause)
| Parameter | Debug | v0.8.0 | Impact |
|---|---|---|---|
fragment_length | 32 | 8 | Affects context but not schedule bug |
S (model width) | 32 | 64 | Wider model needs more training |
r (stability noise) | 1e-3 | 0.0 | Could affect convergence |
compute_lDDT arg order | Swapped in all versions | Swapped | Symmetric metric — doesn’t affect value |
These may explain residual gaps after the fix, but the reversed schedule was the primary cause.
Fix Applied
Same commit pushed to all 10 branches:
fix: reverse EDM sample_schedule direction (sigma_max→sigma_min) and add sigma_data scaling
| Branch | Commit |
|---|---|
main | 341497d |
v0.6.0 | c7e05c0 |
v0.6.1 | 6b7d733 |
v0.6.2 | f43d68c |
v0.6.3 | e733e29 |
v0.7.0 | 8c3497e |
v0.7.1 | 27c5eb1 |
v0.7.2 | b1bcb25 |
v0.7.3 | fabb922 |
v0.8.0 | fe2703f |
Remaining Open Items
compute_lDDTargument order is swapped in all callers (X0passed as predicted,X_hatas ground truth) — the maskMis computed from predicted distances instead of reference distances. Affects local selection behavior but not the reported metric value since|D0 - D|is symmetric.- Fragment length difference (32 vs 8) may need further investigation for optimal lDDT.
- Stability noise
r=0.0in v0.8.0 vsr=1e-3in debug — worth testing.