InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
arXiv:2509.21275v4 Announce Type: replace-cross
Abstract: Long context training is crucial for LLM’s context extension. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (PP) reduces this co…