Kaiden118/RunSleuth-Evidence-Grounded-ML-Training-Failure-Diagnosis-and-Repair-Verification

RunSleuth finds silent bugs in PyTorch training runs, the kind that never crash and barely move accuracy, and accepts a fix only after retraining verifies it. It also checks LLM agents that repair medical-imaging training code: an agent's "fixed" counts only when the evidence agrees.

A ResNet18 on Camelyon17 pathology patches whose classification head was frozen by mistake. Its validation accuracy, 98.2%, is above the healthy run's 97.9%.

runsleuth diagnose --run examples/frozen_head/run_report.json --reference examples/clean/run_report.json --no-llm

Diagnosis: frozen_head  (decided by signature_matcher)
Evidence:
  + head_max_trainable_fraction = 0.0 (== 0)
  + head_max_gradient_fraction = 0.0 (== 0)
  + head_max_update_norm = 0.0 (== 0)
  + backbone_min_update_norm = 0.03682979467213154 (> 0)
  + reference_head_min_trainable_fraction = 1.0 (== 1)
Repair: Set requires_grad=True on the head; the optimizer already contains its tensors.

Its metrics pass every check against the healthy run, and verification still rejects it:

runsleuth verify --candidate examples/frozen_head/run_report.json --reference examples/clean/run_report.json

Decision: rejected
Re-diagnosis: frozen_head
No regression against the reference:
  + id accuracy_drop = -0.003 (<= 0.01)
  + id loss_ratio = 0.8849 (<= 1.1)
  + ood accuracy_drop = 0.005 (<= 0.01)
  + ood loss_ratio = 0.8586 (<= 1.1)
Reason: re-diagnosis against the reference found frozen_head

git clone https://github.com/Kaiden118/RunSleuth-Evidence-Grounded-ML-Training-Failure-Diagnosis-and-Repair-Verification.git
cd RunSleuth-Evidence-Grounded-ML-Training-Failure-Diagnosis-and-Repair-Verification
pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e ".[dev,llm,hf,camelyon]"

Python 3.11. Diagnosis runs on CPU; training needs an NVIDIA GPU with 8 GB. Or use Docker: docker build --target runtime -t runsleuth .

from runsleuth.run_monitor import RunMonitor

monitor = RunMonitor(model, optimizer, "runs/my-run", head="fc")
for epoch in range(epochs):
    with monitor.epoch():
        ...  # your training loop, unchanged
    monitor.log(id_validation_accuracy=accuracy, id_validation_loss=loss)
monitor.close()

With the Hugging Face Trainer, add callbacks=[RunSleuthCallback("runs/my-run", head="classifier")] from runsleuth.hf_callback. Then diagnose, repair, retrain and verify:

runsleuth diagnose --run runs/my-run/run_report.json --reference runs/healthy/run_report.json
runsleuth verify --candidate runs/my-run-fixed/run_report.json --reference runs/healthy/run_report.json

For an LLM explanation, set GEMINI_API_KEY and GEMINI_MODEL, or pass --provider ollama for a local model. It must cite the evidence and cannot change the diagnosis. --no-llm skips it.

*Confirmed only with a healthy reference run; without one RunSleuth waits instead of guessing. †New, checked on one development seed, not part of the results below.

Camelyon17-WILDS, ResNet18 and DeiT-small (Hugging Face), nine injected faults. Held-out seeds were drawn at random after signatures, prompt and thresholds were frozen.

• 11 of 38 faulty runs stayed within the thresholds of their healthy counterparts: validation metrics alone would have passed them.
• LLM explanations agreed with the matcher in 63/64 held-out cases (Gemini 3.5 Flash-Lite) and in all 52 valid replies of a local Qwen3-8B.

Numbers come from the JSON records. Faults were injected by the author and thresholds are development choices; this is not a statistical benchmark.

When an LLM agent repairs a training script and says it is fixed, is it? And does RunSleuth's evidence help the agent, or catch it?

• Tasks: 22 Camelyon17 scripts (ResNet18): 9 faults RunSleuth knows, 7 it does not, 5 correct scripts that look faulty, 1 healthy.
• Conditions: each adds one thing: code only, training runs, raw evidence, diagnosis, a verdict on the agent's own runs.
• Labels: hidden contract tests written from the task statement, independent of RunSleuth.
• Verdict: every final script is retrained, then accepted, reviewed or rejected.

Tasks, conditions, five held-out seeds and the analysis were frozen on 2026-10-13, before the first episode ran. The first model is running; there are no results yet. Agent-edited scripts run in a subprocess under a Python-level guard, not in a container.

python -m runsleuth.agent_tasks list
python -m runsleuth.agent_study run --name <model> --provider gemini
python -m runsleuth.agent_analysis artifacts/agent_study/<model>

☑ done · ◐ in progress · ☐ planned

Set data_dir in configs/camelyon17_clean.json; the download is about 20 GB. Names in angle brackets are run directories that earlier commands print.

python -m runsleuth.camelyon --download --prepare-only
python -m runsleuth.camelyon
python -m runsleuth.camelyon_optimizer_sweep --reference-run <baseline-run> --execute
python -m runsleuth.camelyon_frozen_head_training --reference-run <seed-baseline-run>
python -m runsleuth.camelyon_config_faults --reference-run <seed-baseline-run>
python -c "from transformers import AutoModelForImageClassification as M; M.from_pretrained('facebook/deit-small-patch16-224')"
python -m runsleuth.camelyon_vit --reference-run <seed-baseline-run>
python -m runsleuth.camelyon_loop_faults --reference-run <seed-baseline-run>
python -m runsleuth.heldout draw --count 2
python -m runsleuth.heldout run --reference-run <baseline-run>

python -m pytest runs on CPU with tiny models and synthetic data. GitHub Actions runs it on every push and inside the Docker image.

MIT. Datasets and pretrained weights keep their own licenses.