title: Pomodoro v0.8.0 — Low Validation lDDT Root Cause Analysis tags: [journal, pomodoro, edm, bug, sampling, lddt, v0.8] created: 2026-05-11 updated: 2026-05-11 status: active related:


Pomodoro v0.8.0 — Low Validation lDDT Root Cause Analysis

Problem

Validation lDDT during EDM sampling was ~35, while the debug workspace reference achieved ~60. Training-step lDDT (single-step denoising) was unaffected.

Root Cause

sample_schedule had reversed interpolation direction — ascending (sigma_min → sigma_max) instead of descending (sigma_max → sigma_min). The sampler started from near-zero noise and progressively added noise, then jumped to 0.0 at the final padded step. This inverted the entire denoising trajectory.

Buggy code (all versions v0.6.0–v0.8.0):

# Ascending — WRONG
sigmas = (
    cfg.sigma_min**inv_rho + steps / (num_steps - 1) * (cfg.sigma_max**inv_rho - cfg.sigma_min**inv_rho)
) ** cfg.rho

Corrected code (matching debug reference):

# Descending — CORRECT
sigmas = (
    cfg.sigma_max**inv_rho + steps / (num_steps - 1) * (cfg.sigma_min**inv_rho - cfg.sigma_max**inv_rho)
) ** cfg.rho
sigmas = sigmas * cfg.sigma_data

Secondary Issue: Missing sigma_data Scaling

The debug reference (workspace/debug/edm/debug_training.py) multiplies the schedule by cfg.sigma_data (line 46). All production versions were missing this scaling. Added alongside the direction fix.

Why Training lDDT Was Unaffected

sample_schedule is only called by edm_sample(), which is only invoked in validate(). Training step() uses noise_distribution() for single-step denoising — no schedule involved.

Other Differences Reviewed (Not the Cause)

ParameterDebugv0.8.0Impact
fragment_length328Affects context but not schedule bug
S (model width)3264Wider model needs more training
r (stability noise)1e-30.0Could affect convergence
compute_lDDT arg orderSwapped in all versionsSwappedSymmetric metric — doesn’t affect value

These may explain residual gaps after the fix, but the reversed schedule was the primary cause.

Fix Applied

Same commit pushed to all 10 branches:

fix: reverse EDM sample_schedule direction (sigma_max→sigma_min) and add sigma_data scaling
BranchCommit
main341497d
v0.6.0c7e05c0
v0.6.16b7d733
v0.6.2f43d68c
v0.6.3e733e29
v0.7.08c3497e
v0.7.127c5eb1
v0.7.2b1bcb25
v0.7.3fabb922
v0.8.0fe2703f

Remaining Open Items

  • compute_lDDT argument order is swapped in all callers (X0 passed as predicted, X_hat as ground truth) — the mask M is computed from predicted distances instead of reference distances. Affects local selection behavior but not the reported metric value since |D0 - D| is symmetric.
  • Fragment length difference (32 vs 8) may need further investigation for optimal lDDT.
  • Stability noise r=0.0 in v0.8.0 vs r=1e-3 in debug — worth testing.