Customer Profile
About Meta
Meta’s long-term vision is to build artificial general intelligence (AGI) that is open and built responsibly. To support this ambition, they are aggressively scaling their infrastructure, with a roadmap that includes 350,000 NVIDIA H100 GPUs by the end of 2024.
Recently, Meta unveiled two massive 24,576-GPU data center scale clusters used to train Llama 3. These clusters leverage “Grand Teton” hardware and advanced networks (RoCE and InfiniBand) to push the boundaries of GenAI product development.
The Problem
Interactive Debugging at Scale: With thousands of GPUs, identifying a single problematic node that stalls a training job is incredibly difficult.
Engineers needed a way to perform interactive debugging and ensure that code changes were immediately accessible to all nodes in the environment without compromising the scale required for data loading.
The Solution
Co-developed Parallel NFS: Meta partnered with Hammerspace to build a parallel Network File System (NFS) deployment.
Paired with Meta’s “Tectonic” distributed storage, Hammerspace enables engineers to debug jobs live. Code changes propagate instantly, enabling fast iteration velocity while Tectonic handles the exabyte-scale data loading.
“The combination of our Tectonic distributed storage solution and Hammerspace enable fast iteration velocity without compromising on scale.”