On-Policy Feedback Makes Training Signals Useful

Featured

01 Better End Scores Don’t Teach Denoising

Image-level rewards can judge a diffusion model’s final output. They cannot tell each intermediate denoising prediction how to improve. DiffusionOPSD converts reward gradients into positive and negative targets around the model’s current predictions.

The model learns those explicit targets through self-distillation. Training samples states from trajectories produced by the current policy. Supervision therefore matches the generation paths the model actually visits.

Controlled experiments expose a subtle problem. A theoretically better target does not always deliver a larger gain after one update. Target quality and learnability need separate evaluation. DiffusionOPSD ranked first in 19 of 20 settings across two models and ten evaluators. It beat the strongest competitor by up to 44.0%. Against DiffusionNFT, it cut training GPU hours by 40% and 63%. Full results are still needed to assess image quality, diversity, and transfer across rewards.

Key takeaways:

• Do not optimize only the reward function. Check whether it produces learnable targets for intermediate steps.
• On-policy trajectories reduce mismatch between supervision and real generation paths.
• Evaluate both a target’s theoretical potential and the gain delivered after an actual update.

Source: On-Policy Self-Distillation in Diffusion Models

02 Recallable Memory Can Still Fail in Conversation

A four-month study tracked 40 users, 1,872 sessions, and seven memory conditions. Direct QA accuracy rose from 19.7% to 70.1%, but user satisfaction did not change.

Direct QA asks whether a system can retrieve a fact on demand. Real conversations also require deciding when that fact matters and weaving it naturally into a response. With the same model and context, one system scored 78.8% on direct QA. It cited only 7.9% of relevant facts in real conversations, a gap of nearly 71 percentage points.

Natural memory integration correlated with satisfaction, while direct QA performance did not. That result shows association rather than causation, and the full paper must confirm the controls. Memory evaluations should measure timing, relevance, and conversational fit instead of betting everything on factual recall.

Key takeaways:

• High direct QA scores do not prove that memory improves real user experience.
• Evaluations should test relevance detection and natural integration, not only retrieval.
• Before launch, measure whether memory citations improve satisfaction without becoming forced or unnecessary.

Source: MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

03 Teacher Advice Needs Task-Level Verification

Post-training often relies on two incomplete feedback sources. Verifiable rewards say whether the final answer is correct, but not how each token should change. Online distillation supplies dense token-level guidance, but it can also inherit the teacher’s errors and limits.

OPDVR coordinates both signals on trajectories generated by the current policy. It first rewrites the distillation reward based on whether the full trajectory is correct. ReLU gating then assigns non-negative rewards to correct trajectories and non-positive rewards to incorrect ones.

The teacher distribution still provides dense advice, but task-level verification controls its direction. The method adds no weighting coefficients or switching thresholds. It plugs into policy-gradient methods such as GRPO and consistently beats standard online distillation across six reasoning benchmarks. The abstract does not report the exact gains.

Key takeaways:

• Token-level distillation can fill gaps left by sparse rewards, but final correctness must constrain the teacher’s advice.
• When combining feedback signals, check whether they point in the same direction before tuning loss weights.
• OPDVR adds no new hyperparameters and can plug into GRPO, making it a useful baseline for reasoning pipelines.

Source: On-policy Distillation with Verifiable Reward

04 Preprocessing May Be Your GPU Bottleneck

A pipeline running on GPUs is not necessarily fast end to end. One CPU-only morphology operation can force tensors off the device and back again.

TorchMorph fills that infrastructure gap with 22 operators for batched tensors. It supports up to eight spatial dimensions, including 3D workloads and complex structuring elements often missing from established tools. Its SciPy-like API should require only small changes to existing pipelines.

Against single-threaded SciPy, grayscale morphology reached about 1,100 times higher throughput. Exact Euclidean distance transforms reached about 350 times higher throughput. Those figures do not translate directly into equivalent training gains. Actual benefits depend on tensor sizes and how often the pipeline calls these operations.

Key takeaways:

• Look beyond model kernels when profiling training. Device transfers can erase acceleration gains.
• Vision workloads with batches, 3D data, or complex structuring elements may benefit from keeping morphology on the GPU.
• Validate TorchMorph in the real pipeline because its headline gains use a single-threaded CPU baseline.

Source: TorchMorph: CUDA-accelerated Morphological Transforms