knowledge distillation

topic21 events

papersTODAY 04:00 UTC

Coupled-Noise Distillation Method Targets One-Step Block Generation in Diffusion Language Models

A revised arXiv paper examines why masked diffusion language models can produce incoherent text blocks: they decode every position in a block in parallel from separate marginal distributions. The authors propose a coupled-noise distillation approach intended to let such models generate a whole block in a single step while keeping the tokens mutually consistent. The work is a research preprint and has not been peer-reviewed.

diffusion-models/language-models/distillation/research

papersTODAY 04:00 UTC

arXiv Paper Examines How Much Training Data Matters in On-policy Distillation

A new arXiv preprint investigates how much of the benefit from on-policy distillation actually comes from the training data used. Testing the two teacher-student pairings most often seen in practice, the authors report findings that challenge assumptions about data's role in the method. The work is positioned as a closer look at a technique that has become standard in frontier post-training pipelines.

distillation/post-training/llm/research

papersTODAY 04:00 UTC

Attention Bridge Method Distills Transformers into Mamba Models with Less Data

A new arXiv paper proposes an "attention bridge" technique for converting pretrained Transformer models into Mamba-style state-space models. The approach aims to make the distillation process more data efficient, addressing the high compute cost of training competitive SSMs from scratch. The work targets the gap between the mature Transformer ecosystem and the less developed tooling around state-space architectures.

distillation/ssm/transformers/research

papersTODAY 04:00 UTC

Verifier-Gated Multi-Expert Distillation Aimed at Scientific Reasoning

A new arXiv paper examines multi-teacher on-policy distillation, the technique of training specialist models and then transferring their abilities to a single student using the student's own generated outputs. The authors propose assigning supervision token by token rather than sequence by sequence, with a verifier deciding which expert teacher should guide each token. The method is aimed at scientific reasoning tasks.

distillation/reasoning/multi-expert/arxiv

papersTODAY 04:00 UTC

Learning to Coach: Training an LLM to Distill Guidance From Experience

A new arXiv paper introduces Learning to Coach (L2C), a framework that trains a separate LLM acting as a coach to pull actionable guidance out of experience. The motivation is that raw solution trajectories are typically long and noisy, which limits how well language models can learn from them. The approach aims to convert such trajectories into more useful, condensed coaching signals.

llm/training/arxiv/agents

papersTODAY 04:00 UTC

Counteraction-Aware Multi-Teacher Distillation Aims to Preserve LLM General Skills

A new arXiv paper addresses how domain-specific fine-tuning can erode the general abilities an LLM originally had. The authors propose a counteraction-aware extension of multi-teacher on-policy distillation, which trains on student-generated text under multiple teachers to restore lost capabilities while keeping domain performance. The method targets the trade-off between specialization and retaining broad competence.

distillation/llm/fine-tuning/research

papersTODAY 04:00 UTC

Discrete Beckmann Transport Models Target One-Step Language Generation

A new arXiv preprint introduces discrete Beckmann transport models, a framework for language modeling and reasoning that avoids the multi-step sampling usually required by discrete diffusion and flow approaches. The authors note that existing methods compress sampling steps only by distilling a pretrained autoregressive teacher, which limits the student to the teacher's performance. Their approach instead aims to generate text in a single step without relying on that distillation ceiling.

language-models/diffusion/research/inference

papersTODAY 04:00 UTC

CANAL adds channel-aware noise allocation for private medical image segmentation

A new arXiv paper proposes CANAL, a method that lets hospitals train segmentation models together without sharing patient scans. It uses knowledge distillation to transfer learned feature representations, injecting differential-privacy noise in a channel-aware way so that the privacy budget is spent where it matters most. The approach targets medical image segmentation, where complementary data sits in separate institutions that cannot legally exchange it.

differential-privacy/medical-imaging/knowledge-distillation/federated-learning

papersTODAY 04:00 UTC

FLoKD: Federated Low-Rank LLM Distillation Over Wireless Networks

A new arXiv paper introduces FLoKD, a method for fine-tuning large language models across wireless networks without centralizing user data. It combines federated learning with low-rank adaptation and adaptive knowledge distillation to reduce communication and computation costs. The approach targets privacy-preserving deployment of LLMs in distributed, bandwidth-limited settings.

federated-learning/llm-fine-tuning/knowledge-distillation/privacy

papersTODAY 04:00 UTC

Paper Proposes Adaptive Reciprocal Knowledge Distillation to Preserve Category Correlations

A new arXiv preprint introduces a knowledge distillation method intended to help lightweight student models retain correlation knowledge between categories. The approach, called adaptive reciprocal knowledge distillation, targets the common problem that a large gap in size between teacher and student models weakens knowledge transfer. The work falls within ongoing research on model compression and efficient training.

knowledge-distillation/arxiv/model-compression/research

papersTODAY 04:00 UTC

arXiv Paper Probes Dataset Biases Behind Phantom Transfer

A new preprint on arXiv studies why a teacher model's bias can still pass to a student model even when the training data has had all overt mentions of that bias removed. The authors report that no data-level defense tested so far reliably detects or eliminates this residual, or "phantom," transfer. The work frames the phenomenon as a dataset-level problem rooted in subtle statistical traces rather than explicit labels.

bias/distillation/datasets/safety

papersTODAY 04:00 UTC

Study Examines Training Domain Specialists Without Reasoning Trajectories

A new arXiv paper looks at domain expert distillation, where a teacher model normally passes reasoning traces to a student model. It investigates what happens when specialists are trained only on question-answer pairs and no explicit reasoning supervision is provided. The work aims to clarify how much the reasoning trajectories actually contribute to the student's performance.

distillation/reasoning/fine-tuning/arxiv

papersTODAY 04:00 UTC

Stopping and restarting strategy speeds up multi-turn agentic on-policy distillation

A new arXiv paper addresses the high cost of on-policy distillation, which relies on expensive autoregressive rollouts by the student model and scales poorly when tasks span multiple turns. The authors propose deciding when to halt a rollout and where to resume it, aiming to cut the compute spent on generating student trajectories. The method targets more efficient transfer of capabilities from large teacher models to smaller students in agentic settings.

distillation/agents/efficiency/training

papersSEP 12 04:00 UTC

arXiv paper proposes unified per-token gating family for on-policy distillation

A new arXiv preprint introduces a family of per-token gating methods for on-policy knowledge distillation that mixes forward and reverse KL losses. The authors argue that prior approaches such as EOPD and ToDi each rely on a single fixed gating signal, and their framework generalizes these with multi-channel and bias coefficients. The work is a methodological contribution aimed at improving how distillation losses are weighted per token during training.

distillation/arxiv/training/kl-divergence

papersSEP 10 04:00 UTC

RouteBridge paper introduces bidirectional distillation between NeRF and 3D Gaussian Splatting

A new arXiv preprint presents RouteBridge, a framework for transferring knowledge between neural radiance fields and 3D Gaussian Splatting, two 3D scene representations with complementary strengths. Rather than fixing one representation as the teacher for an entire scene, the method routes distillation bidirectionally, relying on the more reliable representation for each part of the scene.

nerf/gaussian-splatting/distillation/3d-vision

papersSEP 10 04:00 UTC

Paper examines distilling synthetic data for time series foundation models

A new arXiv preprint looks at how time series foundation models are pretrained on artificially generated trajectories where the underlying data-generating process is known. The work focuses on distillation methods rather than the conventional loss-based pretraining objectives that compare model outputs against targets. It aims to improve how these models learn from synthetic time series data.

time-series/foundation-models/synthetic-data/distillation

papersSEP 10 04:00 UTC

Researchers propose MAVEN-T for real-time multi-agent trajectory prediction in autonomous driving

A newly updated arXiv paper introduces MAVEN-T, a method combining reinforcement learning with heterogeneous knowledge distillation to forecast the future paths of multiple agents simultaneously. The work targets real-time deployment in autonomous vehicles, where anticipating surrounding traffic informs collision checking, planning, and control. The approach is designed to remain dependable in dense scenarios with diverse and multimodal agent behaviors.

autonomous-driving/trajectory-prediction/reinforcement-learning/real-time

papersSEP 10 04:00 UTC

Study Examines Decision Shifts and Grounding in Correctness-Gated Multi-Teacher Distillation

A new arXiv paper in cs.AI reports a controlled comparison of eight correctness-gated multi-teacher distillation setups, all sharing the same data sources and a 63.9-million-parameter student model. The authors argue that achieving correct candidate decisions is a different objective from keeping model rationales grounded, and their audit of grounding ended inconclusive. The experiments also documented shifts in decisions and a loss of label functionality across the tested configurations.

distillation/reasoning/evaluation

papersSEP 10 04:00 UTC

CompassOPD adapts on-policy distillation to cross-family model pairs

New research introduces CompassOPD, a method that extends on-policy distillation to settings where the teacher and student models come from different families. It derives within-family likelihood shifts to provide dense token-level supervision on student-generated outputs, tackling the effectiveness drop standard OPD exhibits in cross-family scenarios.

distillation/training/llms

papersSEP 10 04:00 UTC

Zone of Proximal Policy Optimization: teacher guidance via prompts, not gradients

A new arXiv paper argues that knowledge distillation breaks down when the student model is much smaller than its teacher, because imitating the teacher's logits locks the student into its sharpest output modes and harms generalization. The authors propose letting the large teacher guide the small student through prompts during reinforcement-learning fine-tuning instead of through gradient-based distillation. The work appears in the computational linguistics category on arXiv.

distillation/reinforcement-learning/small-models/training

papersSEP 10 04:00 UTC

On-Policy Distillation Proposed for Vision-Language Model Adaptation on Low-Quality Data

A new arXiv paper introduces an on-policy distillation approach for adapting compact vision-language models from a larger task-trained teacher. Rather than relying solely on teacher predictions as training targets, the method lets the student learn from its own outputs, which the authors report makes it especially effective when multimodal training data is noisy or low quality.

distillation/vision-language/multimodal/efficiency