By Eric Bassier, Senior Director, Solution Marketing
Late last week, Meta released Llama 3, its latest open source large language model, with models of 8B and 70B parameters now available. In its April 18 blog post, Meta discusses how it improved Llama 3 by scaling up pre-training, along with other factors. In highlighting its training approach and infrastructure, Meta noted:
“Our most efficient implementation achieves a compute utilization of over 400 TFLOPS per GPU when trained on 16K GPUs simultaneously. We performed training runs on two custom-built 24K GPU clusters.”
Meta previously provided more detail on these “custom-built 24K GPU clusters” in a separate blog published on March 18. That blog highlighted Meta’s use of Hammerspace as part of its AI training infrastructure, and stated this:
“Among other benefits, Hammerspace enables engineers to perform interactive debugging for jobs using thousands of GPUs as code changes are immediately accessible to all nodes within the environment. When paired together, the combination of our Tectonic distributed storage solution and Hammerspace enable fast iteration velocity without compromising on scale.”
If you are interested in learning more about how we are working with Meta, please contact us.
