Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

T. Chen, A. Kubicek, L. Huang, a. Torsten Hoefler:

 CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training

(In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference, presented in Boston, MA, USA, pages 1089-1108, USENIX Association, ISBN: 978-1-939133-48-9, Jul. 2025)

Publisher Reference

Abstract

Training large language models (LLMs) now requires resources that exceed a single datacenter, making cross-datacenter strategies increasingly crucial. We present CrossPipe, a framework designed to optimize model training across geographically distributed datacenters by explicitly modeling and mitigating the impact of network latency and limited bandwidth. It enables unified analysis and optimization incorporating both pipeline parallelism (PP) and opportunities for overlapping data parallelism (DP) communication. CrossPipe generates optimized pipeline schedules using either solver-based optimal or fast near-optimal greedy algorithms, built upon a flexible execution engine that separates scheduling logic from communication details. Our evaluation shows that CrossPipe reduces training time by up to 33.6% compared to traditional pipeline schedules under identical memory constraints. When memory constraints are relaxed, CrossPipe maintains strong performance despite communication delays, approaching the efficiency of idealized schedules without delays. CrossPipe offers improved scalability and resource utilization, particularly in environments with high network latency or limited bandwidth.

Documents

download article:
download slides:


Recorded talk (best effort)

 

BibTeX

@inproceedings{tiancheng2025crosspipe,
  author={Tiancheng Chen and Ales Kubicek and Langwen Huang and and Torsten Hoefler},
  title={{CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training}},
  year={2025},
  month={07},
  pages={1089-1108},
  booktitle={Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference},
  location={Boston, MA, USA},
  publisher={USENIX Association},
  isbn={978-1-939133-48-9},
  doi={10.5555/3768039.3768103},
}