A study argues that one-shot magnitude pruning does more than compress neural networks, acting as an implicit regularizer for automatic speech recognition. Testing with Whisper-small, the authors combine gradient- and Fisher-based sensitivity measures to guide which weights to remove. The work reframes pruning as a training technique rather than only a efficiency tool.
Researchers propose X-AuT, a method that progressively compresses the audio encoder of speech large language models rather than deleting whole blocks at once. Because abrupt block removal distorts the embeddings the decoder receives and leads to word deletion and premature end-of-sequence errors, the approach uses cross-scale distillation to shrink encoder depth while preserving output quality. The aim is to cut inference cost without the accuracy loss typical of standard pruning.
A new paper examines how to shrink Mixture-of-Experts language models by pruning experts, targeting models specialized for particular domains. The authors find that earlier compression techniques break down badly when the trimmed models are tested on general-purpose benchmarks. The work aims to make these large, sparsely activated models easier to deploy without losing performance.