papersSEP 10 04:00 UTC
X-CoSD: Cross-Vocabulary Collaborative Speculative Decoding for Efficient LLM Inference
Researchers introduce X-CoSD, a distributed inference framework in which a small on-device model drafts tokens and a larger server-side LLM verifies them. The method tackles the mismatch between the two models' vocabularies while cutting the communication cost between device and server. The work aims to make collaborative speculative decoding practical for latency-sensitive edge deployments.