By Floyd Christofferson and Molly Presley
At Hammerspace, we don’t just follow trends—we create them. Our ongoing contributions to the open source Linux kernel are not just incremental updates; they’re paradigm-shifting innovations. With the latest advancements to the kernel, Hammerspace has enabled standard Linux servers to become high-performance engines for enterprise IT, AI, and HPC storage. By tightly integrating NFSv4.2’s potential with our parallel global file system, Hammerspace has enhanced Linux to combine the standards-based compatibility needed in the enterprise with HPC-class performance needed for AI/DL use cases. In doing so, we’ve shattered the limitations of proprietary storage systems to address the extreme performance requirements of GPU computing and AI.
Redefining Performance with Recent Additions to the Open Source Linux Kernel
- Massive scalability and blazing-fast throughput.
Thanks to Parallel NFSv4.2 with FlexFiles and N-Connect, Hammerspace delivers unprecedented parallelism, transforming how data flows across distributed servers even at extreme performance levels. - Instant responsiveness for metadata-heavy workloads.
Attribute Delegations allow clients to intelligently cache metadata locally. The result? Lightning-fast access and zero unnecessary server chatter. - Resilient by design.
With Fast Failover, Hammerspace adds resilience to ensure workflows stay alive—even if individual servers go offline. - Blazing speed data delivery, direct to compute.
LOCALIO (NFS Protocol Bypass) eliminates unnecessary overhead when server node storage is on the same host, bypassing the network RPC protocols to send data straight to GPU/CPUs without the latency of the network loopback as was previously needed.
How Hammerspace Leverages These Game-Changers
The recent additions contributed by Hammerspace to the standard kernel not only enhance the capabilities of Linux, but also its position as the backbone for secure, scalable, and high-performance storage environments tailored to the demands of today’s data-driven world. And tight integration with these Linux features is also the backbone of how Hammerspace operates. Not only does this ensure vendor-neutrality by being compatible with all standard Linux distributions, but Hammerspace builds on these kernel advancements to turbocharge them with additional functionality, creating a unified parallel global file system with automated data orchestration that spans storage silos, multiple sites, and cloud providers.
This tight integration between Hammerspace and standard Linux also means customers don’t need to install any proprietary client or server-side software to take advantage of these innovations, even within existing storage and server environments from any vendor. Together these capabilities make data truly unbound by vendor silo limitations.
Here’s what that looks like:
- When paired together, pNFS v4.2 with FlexFiles and N-Connect enable high-performance parallel access to data by distributing file operations across multiple storage nodes. This design allows clients to perform simultaneous read/write operations across a cluster of servers, delivering extreme parallelism for high-throughput workloads. Combined with N-Connect, which allows multiple parallel connections between the client and the server, these innovations overcome the challenges of TCP/IP networking and get closer to the throughput that was previously only available with RDMA networking (RoCE and Infiniband). With N-Connect you can get 90+ percent of the performance of RDMA without the complexity and expense of RDMA networking. When leveraged in connection with a parallel file system, a standards-based, high-performance data architecture such as is needed for AI and other HPC applications becomes reality, even in existing enterprise IT environments.
The contribution into the kernel of these Linux patches optimizes network utilization by balancing data transfer loads across all available paths, especially in extremely large environments. The result is the ability to scale linearly from small use cases to the largest hyperscale environments, and is ideal for AI training, HPC workloads, and other performance-intensive tasks.
By eliminating the proprietary bottlenecks often associated with specialized hardware or custom protocols in other parallel file systems, pNFS v4.2 with FlexFiles and N-Connect delivers the speed, flexibility, and reliability needed for modern workloads. It empowers organizations to confidently scale their existing infrastructure while maintaining efficiency and performance at every level of operation.
- Attribute Delegations allow decentralized data to behave as if it’s local by enabling clients to intelligently cache metadata, eliminating the crazy number of GETATTR requests previously needed. This enhancement significantly reduces the latency normally caused when accessing and managing data across a unified global namespace of different storage types. The result? Applications no longer need to wait for constant back-and-forth communication with servers—they just work. For latency-sensitive use cases where responsiveness is critical, such as real-time analytics or interactive simulations, Attribute Delegations ensure seamless performance no matter where the data resides.
- Fast Failover redefines resilience by ensuring uninterrupted workflows even in the event of server failures. This feature enables client applications to seamlessly re-route workloads to another available server without experiencing downtime or data loss. The era of unexpected disruptions and costly delays is over—workloads keep running, dynamically moving as needed to ensure productivity and reliability even in the most demanding scenarios.
- LOCALIO revolutionizes I/O performance by eliminating unnecessary NFS network RPC calls for read, write, and commit operations when the client and server share the same host. LOCALIO adds intelligence to the client, enabling it to recognize that the NFSv3 data server is co-located inside the same host kernel. In this way, LOCALIO bypasses the networking layers within the host kernel, which eliminates the unnecessary network loop back and increases performance by further reducing latency. This avoids traditional I/O bottlenecks, delivering performance levels that conventional storage architectures can’t touch.
When paired with Hammerspace’s automated global data orchestration, LOCALIO activates stranded NVMe storage on GPU servers, turning these underused resources into ultra-fast storage as an active Tier 0. Data on this storage is protected within the Hammerspace data environment, with automated mirroring across other GPU servers or non-disruptive tiering to any other type of external storage. Activating this capability accelerates operations such as checkpoints to increase GPU utilization while keeping the data protected within the global file system. For high-performance AI and HPC workflows, LOCALIO enables peak efficiency and speed for even the most performance-intensive tasks.
Hammerspace doesn’t just use Linux—it transforms it and has tightly integrated the Hammerspace Global Data Platform software to take full advantage of the kernel features it has contributed. This innovation continues with the next wave of Linux storage advancements that Hammerspace is pushing upstream into the community, which extend the boundaries even further.
These include:
- Client-Side Erasure Coding shifts the responsibility for redundancy calculations from servers to clients. This smarter approach not only reduces the load on storage servers—freeing up resources for other critical operations—but also enables more efficient scaling in distributed environments. For workloads requiring massive parallel writes, such as AI training or high-frequency trading simulations, this capability ensures robust data protection without bottlenecks, allowing for faster performance at any scale.
- Striped FlexFiles is a Linux enhancement that takes parallelism to the next level by simultaneously distributing data across multiple servers. This feature extends the use of pNFS v4.2 beyond traditional file-sharing use cases to accelerate the most demanding workloads. It ensures that even applications requiring extreme performance needed in HPC—such as genomic sequencing or CFD simulations—can benefit from enhanced speed and efficiency with the standard parallel file system included in the Linux kernel. By widening the scope of applications pNFS can address, Striped FlexFiles makes the protocol a powerful tool for both modern and traditional HPC environments.
- TLS Connection for Linux RPC will introduce enterprise-grade encryption for secure data transport across both local and remote connections. Unlike traditional encryption methods that often trade speed for security, this enhancement is designed to maintain high performance while safeguarding sensitive information. For organizations handling distributed datasets or operating in regulated industries, this ensures compliance without compromising the speed and efficiency of their workflows.
Why This Matters
Traditionally, HPC-class Mod/Sim use cases have relied on the types of parallel file system technology that most enterprise IT teams cannot easily support. The problem for enterprises who need HPC-levels of performance to power new GPU-computing workloads is that HPC file systems such as Lustre, BeeGFS, and others become yet another data silo that requires proprietary client software to utilize and manage. In addition, such parallel file systems do not have high-performance, multi-protocol SMB, NFS, and S3 access without routing through bottlenecked gateways. Being non-standard, they also have limitations on which storage and server types they support; plus they typically require InfiniBand or other expensive RDMA networking to achieve high performance.
For enterprise IT teams who often lack specialized expertise in HPC technologies, these factors have been blockers, preventing widespread adoption of HPC-class parallel file systems in enterprise data centers.
The problem for enterprises considering AI and other modern high-performance use cases is that enterprise IT environments cannot easily accommodate the additional overhead of an HPC parallel file system on top of the existing scale-out NAS systems they use today. This makes the ROI calculation very challenging for enterprise AI workloads, which increasingly need to feed GPU-clusters at HPC-class performance levels.
So enterprise IT planners are faced with questions such as: “Do we need to build another high-performance storage silo just for AI use cases?”
It turns out that lurking within the Linux kernel in every server and storage device in every data center on earth is Parallel NFSv4.2 with FlexFiles, a standards-based high-performance parallel file system with the performance and scalability needed to solve this problem.
As a standards-based solution included in all standard distributions, pNFSv4.2 with FlexFiles already runs on enterprise-class storage and commodity hardware from any vendor, requires no proprietary client software, and doesn’t even need expensive HPC-style networking to achieve the extreme performance needed for even hyperscale AI use cases, such as Meta’s AI Research SuperCluster.
Tight integration with the Linux community is a key component of Hammerspace DNA. Trond Myklebust, Hammerspace’s CTO, has been the Linux NFS Client kernel maintainer for more than 20 years. Hammerspace ‘s Senior Principal Software Engineer Mike Snitzer is also a Linux kernel maintainer, responsible for the upstream Linux kernel’s Device Mapper (DM) subsystem.
The result is that in the last 10 years Hammerspace engineers have submitted roughly 2,400 enhancements and patches to open source Linux projects in just the NFS and kNFSd codebases alone. These contributions adhere strictly to IETF internet protocol standards and often play a role in shaping their evolution.
Importantly, this commitment to standards also means that key differentiated features within Hammerspace software are tightly integrated with the high-performance capabilities we’ve submitted into the Linux kernel. Hammerspace takes advantage of this tight integration to create its Parallel Global File System that spans multiple sites and clouds in a unified global namespace, plus the automated, non-disruptive data orchestration capabilities to ensure data is always where it needs to be across any storage type or location. In addition, Hammerspace features such as data-in-place assimilation of metadata from existing storage, high-performance standard SMB, NFS, & S3 client-side protocols, and much more are all unique Hammerspace capabilities that benefit from tight integration with our contributions to the standard Linux kernel.
The result is that HPC and enterprise customers can use Hammerspace to achieve HPC-class performance without sacrificing enterprise-class ease of use and RAS features they expect in scale-out NAS platforms. This also means both enterprise and HPC environments can leverage their existing storage and infrastructure to achieve these performance levels without installing proprietary client software or wrangling data across siloed incompatible systems.
We are entering the next data cycle in which high-performance AI/DL use cases are only just beginning to discover the untapped potential buried within unstructured data. And with its contributions to advance high-performance features in standard Linux, Hammerspace is unlocking it at a level that’s never been possible. Linux is the predominant OS in both enterprise and HPC data environments, and with these advancements it is now ready to tackle the most demanding enterprise and AI workloads.
Hammerspace is the platform making it happen.
Want to learn more?
Check out the pNFS Resource Center and listen to the webinar with Linux kernel maintainers, Trond Myklebust and Mike Snitzer, “New Advancements in PNFS / NFS v4.2 for High Performance and Distributed Storage”.
