From the MTIA v1 hardware to the recently unveiled Llama 2 and Meta’s cutting-edge large language model, the tech giant is setting the stage for the future of artificial intelligence across all levels.
The advancement to a next-generation system is imperative for Meta to deliver AI products and services at its current scale.
The 2023 edition of Networking at Scale delved into various aspects such as performance workloads, expansive GenAI models, and networking systems developed by Meta’s engineers and researchers in recent years. Topics covered include load modeling, performance optimization, debugging, benchmarking, routing strategies, weight balancing solutions, and network design, both physical and logical. Anticipating the evolving needs of GenAI models is also on the horizon.
Facilitating Inference Clusters and GenAI Training
At Meta, the development of new GenAI systems and their integration into product features is a top priority. However, the complexity and scale of GenAI models pose challenges for Meta’s network infrastructure.
Jongsoo Park and Petr Lapukhov explore the unique requirements of these large language models and how Meta’s infrastructure is adapting to support the new GenAI environment.
Meta’s Network Evolution to Enable AI
As AI workloads continue to grow, Meta’s AI infrastructure has transitioned from CPU-based to GPU-based training. To support these evolving techniques and workloads, Meta has implemented large-scale distributed network-interconnected systems.
Currently, the training models at Meta utilize a RoCE-based network fabric with a CLOS topology, where back switches provide Scale-Out connectivity to the cluster’s GPUs, and leaf switches are connected to GPU hosts.
Hany Morsy and Susana Contrera delve into the evolution of Meta’s network infrastructure to meet the demands of AI services. They discuss the challenges faced, innovative approaches taken, and corporate considerations that contributed to the development of Meta’s high-performance network fabric for AI workloads.
Scaling RoCE Networks for AI Training
Adi Gangidi provides an overview of Meta’s RoCEV2 transport-based RDMA implementation to support the production AI training system. He details how Meta’s infrastructure was designed to optimize reliability and performance crucial for AI-related workloads.
The discussion also touches upon future development opportunities and the challenges overcome in route, transport, and hardware layers to elevate Meta’s infrastructure.
Networks for AI Training in Traffic Engineering
Since 2020, Meta has been operating distributed training clusters based on RoCE to manage its internal AI training workloads. However, early challenges in maintaining consistent performance were encountered.
Shuqiang Zhang and Jingyi Yang shed light on one of Meta’s solutions to this issue, the consolidated customer architecture, which automatically distributes traffic across all relevant paths in a load-balanced manner. They discuss the design, development, evaluation, and operational experience of this consolidated traffic engineering solution.
Network Observability for AI/HPC Training Workflows
Enabling and scaling Meta’s AI training and inference workloads hinges on high-performance, reliable communication over the AI-Zone RDMA network. Capturing top-down observability from workload to network communication is essential to attribute performance regressions and training failures to the backend network when necessary.
Meta has introduced crucial tools like ROCET and PARAM metrics, along with the Chakra framework, to facilitate this process. Shengbao Zheng elaborates on the utilization and design contexts of these tools in this discussion.
Arcadia: Enhancing AI System Performance Simulation
Arcadia, an end-to-end AI system performance simulator, is introduced to assess network, memory, and computational performance in AI training clusters. This unified approach aids in decision-making processes and advances the development of AI systems by providing a comprehensive performance analysis framework.
Zhaodong Wang and Satyajeet Singh Ahuja elaborate on how Arcadia empowers Meta’s engineers to evaluate the performance impact of administrative tasks on production AI models and make informed decisions during routine operations. The tool’s potential contributions to the evolution of AI techniques and hardware are also highlighted.
