Hammerspace Introduces an All-New Storage Architecture | Read the Press Release

Insight Into Meta’s Llama 3 Training Environment

By Eric Bassier, Senior Director, Solution Marketing

Late last week, Meta released Llama 3, its latest open source large language model, with models of 8B and 70B parameters now available. In its April 18 blog post, Meta discusses how it improved Llama 3 by scaling up pre-training, along with other factors. In highlighting its training approach and infrastructure, Meta noted:

“Our most efficient implementation achieves a compute utilization of over 400 TFLOPS per GPU when trained on 16K GPUs simultaneously. We performed training runs on two custom-built 24K GPU clusters.”

Meta previously provided more detail on these “custom-built 24K GPU clusters” in a separate blog published on March 18. That blog highlighted Meta’s use of Hammerspace as part of its AI training infrastructure, and stated this:

“Among other benefits, Hammerspace enables engineers to perform interactive debugging for jobs using thousands of GPUs as code changes are immediately accessible to all nodes within the environment. When paired together, the combination of our Tectonic distributed storage solution and Hammerspace enable fast iteration velocity without compromising on scale.”

If you are interested in learning more about how we are working with Meta, please contact us.

About the Author

Eric is the Senior Director, Solution Marketing and Sales Enablement, at Hammerspace. He is an innovative product leader with extensive experience launching and evangelizing products, driving go-to-market strategies, and creating compelling content to drive customer engagement and growth.