Concertina: Adaptive Pipeline Parallelism for Long-Context LLM Training
A new arXiv paper introduces Concertina, a data-centric approach to pipeline parallelism aimed at making long-context LLM training more efficient. The authors argue that sequence parallelism carries heavy communication costs, and that pipeline parallelism's benefits depend on how the model and data are partitioned. Concertina adaptively splits pipeline stages to better handle long sequences on heterogeneous hardware.