Tech Podcast Brief - How HPC & AI are Changing DC Networks

Source: Heavy Networking podcast episode with Rob Sherwood, discussing the impact of High-Performance Computing (HPC) and Artificial Intelligence (AI) on data center network design.

Main Themes:

• The unique demands of AI training workloads necessitate dedicated network infrastructure.
• Traditional networking assumptions like oversubscription and best-effort delivery do not apply to HPC and AI.
• Bandwidth, power, and cooling are major challenges that require innovative solutions.
• The network interface card (NIC) architecture is evolving to address these challenges, with a shift towards smarter NICs, RDMA, and even optical interconnects.

Most Important Ideas/Facts:

1. Collective Communication:

• AI training, especially building large language models (LLMs), relies on collective communication operations like "all-reduce", where data is exchanged and processed simultaneously across all nodes.
• Traditional unicast-based networks are ill-suited for this, as they lead to congestion, packet loss, and performance degradation.
• "Collective Communications is everybody does the same thing at once in lock step... it is the most bursty thing that you could create on a network where everybody in lockstep is sending all the packets all at once to one place."
• Dedicated AI training networks are the current best practice to ensure optimal performance.

2. Power and Cooling:

• Power consumption of HPC and AI workloads is massive and rapidly increasing.
• "At 800 gig it becomes 50% of the total power spend is your network."
• This poses challenges for data center operators in terms of:
• Power provisioning: Securing sufficient power from utilities is a major hurdle.
• Cooling: Removing the generated heat requires significant resources and innovative cooling technologies.
• Sustainable solutions are crucial to mitigate the environmental impact.

3. Bandwidth:

• AI training demands immense bandwidth due to the high volume of data exchanged between nodes.
• Oversubscription is not an option, requiring full bisectional bandwidth.
• "In an AI training Network you cannot have over subscription you must be you know full full box bisectional bandwidth... one to one over subscription if you want to think of it that way."
• Optical interconnects are gaining traction due to their power efficiency and ability to handle high bandwidth.
• "Anything above 400 gigabit you really start to get bitten by the power cost of the network... I'm seeing designs where it's actually Optical down to the servers."

4. NIC Architecture:

• Traditional "dumb" NICs are insufficient for HPC and AI, as they lack the capabilities for efficient data transfer and offloading.
• RDMA (Remote Direct Memory Access) is critical for bypassing the CPU and transferring data directly between memory and the network.
• Smart NICs (DPUs/IPUs) with specialized processing capabilities are increasingly used for offloading tasks and accelerating network operations.
• "The dumb NICs are actually quite smart... to qualify you mean... I can offload some pretty fantastic stuff these days..."
• Integration of NICs with GPUs is being explored to further reduce latency and bottlenecks.
• "What ends up happening as data comes in off the NIC is by RDMA it will write the the packet memory directly into the GPU Ram."

Recommendations for Enterprises:

• For infrequent or experimental AI training, cloud providers offer the most convenient and cost-effective solution.
• For more dedicated use, consider pre-integrated commercial off-the-shelf systems tailored for AI training.
• Seek expertise in HPC networking to navigate the complexities of building and operating such infrastructure.
• Be aware of the rapidly evolving landscape and stay informed about emerging technologies like Ultra Ethernet and Optical circuit switches.

Investment Opportunities:

• Power-efficient solutions, including Optical circuit switches, Advanced flywheel designs, and Load shifting technologies, offer significant potential.
• Liquid cooling technologies with improved thermal properties and operational safety are another promising area.

Overall Takeaways:

• The rise of HPC and AI is driving fundamental changes in data center network design.
• Bandwidth, power, and cooling are key considerations that demand innovative solutions.
• The networking industry is actively developing new technologies and standards to address these challenges.
• Enterprises and investors need to be aware of this evolving landscape and make informed decisions about their infrastructure investments.