Less is MoE: Trimming Experts in Domain-Specialist Language Models

A new paper examines how to shrink Mixture-of-Experts language models by pruning experts, targeting models specialized for particular domains. The authors find that earlier compression techniques break down badly when the trimmed models are tested on general-purpose benchmarks. The work aims to make these large, sparsely activated models easier to deploy without losing performance.