By Floyd Christofferson
Introduction: Meeting the Data Demands of AI, HPC and Deep Learning
For decades, enterprise IT departments have been in a race against time. According to industry research, about 90% of all enterprise data is unstructured, and studies show that often less than 10% of such file data remains in active, high-performance storage 90 days after it was created.
Although data types differ greatly between industries, IT staff across them all wrestle with the “time-value” of data when trying to figure out how to keep up with data growth. That is, IT pros must decide which data they should keep, which can be safely discarded, which storage tier the different classes of data should be saved onto, when to do so, and for how long. Does the data need to be immediately accessible? Or can it be gradually shuffled downstream to less expensive and lower performing storage tiers, until eventually it winds up in a deep cold archive somewhere?
Since the early 1990s, data management technologies to enable Information Lifecycle Management (ILM) have attempted to bring structure to unstructured data and figure out a way to automatically reconcile the true time-value of data with the storage it lives on. The problem is, as data volumes increase so do storage costs. Storage is a consumable IT resource that costs both CAPEX and OPEX. When data is cold and stale, those costs are even more painful and rather than adding value, data becomes a cost center.
The recent explosion of artificial intelligence (AI) and deep learning (DL) technologies has turned the conventional ILM-style time-value equation on its head, with AI/DL technologies promising to extract new value from otherwise stale unstructured data. The difficulty is that the GPU clusters needed for AI applications require data to be staged on high-performance storage. Even the best enterprise NAS storage solutions from all vendors fall far short of this requirement because of their inherent performance bottlenecks and scaling limitations.
Organizations are left wondering how to meet the AI-driven performance requirements without overhauling their existing infrastructure or adding new storage silos, which would inevitably increase the costs of their storage infrastructure even more. Will AI use cases even deliver enough value to justify this additional expense?
The Enterprise IT Performance Dilemma
High Performance Computing (HPC) environments have always needed storage platforms that can scale to the extremes needed for supercomputers. Parallel file systems such as Lustre, GPFS, and others were tailor-made for such environments. But while they are extremely fast, the price for such performance is that parallel file systems are not easy to use, require special client software, don’t support standard file protocols or enterprise data services, plus they require exotic high-performance networking, and specialized hardware.
Enterprise data environments cannot easily accommodate the additional overhead of HPC parallel file systems on top of the typical enterprise NAS systems they use today. This makes the ROI calculation very challenging for enterprise AI workloads that need to feed GPU-clusters with HPC-class performance.
So IT planners are faced with questions such as: “Do we need to build another high-performance storage silo just for AI use cases? Which data should we move into this new repository? How do we do this without duplicating infrastructure and creating file copy sprawl, or without abandoning our existing investments in traditional enterprise NAS storage, and especially without interrupting users and applications?”
It turns out that lurking within the Linux kernel in every server and storage device in every data center on earth is a standards-based high-performance parallel file system. It uses open standards, can run on existing storage and commodity hardware, requires no proprietary client software, and doesn’t even need special HPC-style networking.
Leveraging Standard Protocols to Achieve Greater Performance with pNFS v4.2
Parallel NFS (pNFS) v4.2, is a time-tested solution that has the advantage of leveraging existing infrastructure to create a high-performance parallel file system, and which has been included in the Linux kernel since 2019. Unlike its predecessor, pNFS v4.1 which struggled with severe performance issues, pNFS v4.2 with Flex Files enables the linear scalability needed for AI/DL workloads, even to the point where it can accelerate existing legacy scale-out NAS storage systems.
In fact, Hammerspace software uniquely leverages the parallelism of pNFS v4.2 to automate AI workloads with GPU data orchestration at extreme performance levels. It is so powerful that it achieves the linear scalability needed by Meta, for example, to feed 24,000 GPUs in its AI Research SuperCluster at an eye-watering 12.5TB per second to power the hyperscaler’s Llama 2 & 3 large language models (LLMs).
Hammerspace does this with standard pNFS v4.2 with Flex Files, running on Meta’s existing commodity storage/server infrastructure and standard Ethernet, and without the need for proprietary client software or alterations to existing storage.
Below is a summary of the key capabilities of pNFS v4.2 with Flex Files that make it ideal for AI/DL workloads.
For a deeper dive into pNFS v4.2 with Flex Files and how Hammerspace has tightly integrated pNFS into its Global Data Platform software, please check out a new Hammerspace white paper called: How to Adapt Legacy Infrastructure to Power GPU-Based AL/DL Workloads Using Parallel NFS v4.2
Why pNFS v4.2 is Ideal for GPU-based AI/DL Workloads
1. Separation of Metadata and Data Paths
Traditional NAS storage systems from all vendors suffer from a fatal bottleneck to performance scalability because both metadata and data are funneled down the same network path.
With pNFS v4.2 the data and metadata paths are separated, allowing direct data access between client and storage without bottlenecking through a head node or controller. This reduces latency and enables almost limitless linear scalability, such that pNFS v4.2 systems can scale to over thousands of storage nodes without the performance degradation seen in even the best scale-out NAS platforms available today.
Fig. 1: Parallel NFS v4.2 enables linear scalability by separating the metadata control plane, enabling parallelized direct I/O between clients and data servers.
2. Client-Side Intelligence
The pNFS v4.2 client, which is available by default in all Linux distributions, introduces client-side intelligence that caches file state metadata. This significantly reduces the overhead of constant metadata requests or “chattiness” between client and server that is needed in NFS, SMB and other enterprise file systems. The pNFSv4.2 telemetry feed-back loop between client and server also enables the system to automatically adapt in real time to performance issues that may occur on the network, giving it a self-healing capability that is unheard of in typical NAS systems. And since the pNFS v4.2 client is already in the Linux kernel, enterprise IT staff do not need to alter existing application servers or user access. Everything is standards-based, exactly as though it were a standard enterprise NAS.
3. N-connect Capability
This feature breaks open the TCP/IP bottleneck, allowing multiple network connections to a single storage node. This means I/O loads can be fanned out in parallel across multiple network interfaces while still using normal Ethernet. This overcomes the problem in traditional TCP/IP-based enterprise networks which only support a single connection, which limits scale-out performance. With pNFSv4.2 such I/O load balancing and parallelism would otherwise require IT teams to add the additional expense of RDMA or InfiniBand networks to their infrastructure.
4. Compatibility with Existing Infrastructure
Unlike bespoke high-performance storage solutions that need vendor-specific hardware, pNFS v4.2 is compatible with any commodity servers and storage, enabling parallel file system performance across any combination of existing storage via the ubiquitous NFSv3 access protocol (See Fig. 1). Data is still routed directly between client and server, bypassing controller bottlenecks, which makes pNFS v4.2 compatible with any existing enterprise storage platform. This means organizations can upgrade portions of their data environment to enable high-performance parallel file system performance without overhauling their current storage infrastructure.
5. pNFS v4.2 Flex Files Layouts
Flex Files means that when a pNFS v4.2 client asks for data, the metadata server provides a real-time layout, or map, of exact file locations, even when the data set might be fragmented across multiple back-end storage systems. This capability leverages the client-side intelligence noted above. As soon as the client receives the layout, the client can parallelize I/O across multiple paths directly to the storage location, reducing latency and dramatically increasing performance. This metadata-driven performance boost helps IT staff keep GPU clusters fully utilized, and also enables a seamless transition from one job to the next without interruption.
6. Live Data Mobility
Because the pNFS v4.2 metadata control plane is separated from the I/O path, pNFS v4.2 can create new instantiations of active files on different storage nodes in the background, even when files are open and actively being read or written. In this way, staging data between tiers, or into an NVMe array for extreme performance needed to feed a GPU cluster can be done from existing storage in the background even on live files, without interruption to users or applications. This eliminates the need to retool data centers or disrupt existing operations to begin experimenting with AI use cases.
Hammerspace builds on pNFS v4.2 in its Global Data Platform
Hammerspace has unique expertise with pNFS v4.2, and has been contributing many of its key features into the Linux community for years. As such, Hammerspace Global Data Platform software is designed as a vendor-neutral solution, for maximum compatibility with existing IT environments. In fact, Hammerspace’s Hyperscale NAS architecture is built upon pNFS v4.2 with Flex Files, and is included in Hammerspace software as a standard capability.
In addition to that, below are some of the key features included in Hammerspace Global Data Platform software that also take advantage of pNFS v4.2:
- Data-in-Place Metadata Assimilation: This capability means Hammerspace can rapidly incorporate metadata from data in place on any existing storage, enabling users to get immediate file access without needing to wait for data to be migrated to a new storage platform.
- Parallel Global File System: Leveraging the file system metadata it has assimilated from existing storage, Hammerspace uses it to present users with a high-performance Parallel Global File System that spans all storage types to provide global file access via standard SMB, NFS and S3 protocols. In this way, Hammerspace-powered data environments can not only bridge incompatible storage silos in a single data center, but it can also create a global namespace that unifies high-performance file access across multiple sites, cloud providers and regions.
- Automated Data Orchestration: Hammerspace includes a powerful objectives-based policy engine that leverages multiple metadata types to trigger data placement, tiering, migration, and of course workflow provisioning for GPU-based computing. This data orchestration capability can even utilize custom metadata as a variable to automate data placement and movement non-disruptively, ensuring seamless AI/DL workflow integration in the background without interruption to existing user workflows or the need to create file copy sprawl into net-new storage repositories.
In summary, pNFS v4.2 with Flex Files brings to standard enterprise IT data center environments the HPC-class parallel file system performance needed for AI/DL workloads. In this way, enterprises in all industries can begin the AI journey with little alteration to their existing infrastructure, and without the huge investment and operational complexity needed to build new AI-specific high-performance storage silos.
By eliminating traditional bottlenecks and enhancing scalability and performance over even the best enterprise scale-out NAS platforms, pNFS v4.2 with Flex Files provides a robust, standards-based solution that brings HPC-class performance to standard enterprise data center infrastructures. It enables global file access via standard protocols, to meet the demands of modern GPU-intensive AI/DL applications of any size.
