Opens in a new tab
Contact Us
Get Started

The GPU Server Local NVMe Audit: Mapping Stranded Capacity, Checkpoints, and Idle GPUs Before You Buy More Flash

Many teams keep buying external flash while fast NVMe drives sit stranded inside their GPU servers, invisible to every other node. GPU server local NVMe can serve as shared, cluster-visible storage when software exposes it as a policy-protected tier. Only your own workload data shows whether it should. Hammerspace Tier 0 is built to expose that local NVMe as a tier, and this audit shows you how to decide whether doing so fits your cluster.

Why Is GPU Server Local NVMe Stranded While Flash Spending Grows?

Stranded NVMe capacity is a utilization problem, not a capacity problem. The drives exist, but each one is visible only to its own node. Any job that needs shared data goes to external storage instead.

GPU servers commonly ship with local NVMe, and so do many cloud GPU instances. In practice, those drives end up as scratch space, a dataset cache, or nothing at all. Meanwhile, external flash keeps growing because checkpoints, shared datasets, and model weights must be readable by every rank or replica.

Two very different things share the same name here:

  • Untreated local NVMe: a per-node filesystem with node-contained visibility and no protection. Its contents disappear with the node.
  • Software-defined Tier 0: the same physical devices, exposed into a shared namespace, with placement and protection governed by policy.

Both use identical hardware. What separates them is access semantics and protection. If you need the fundamentals first, read why Tier 0 changes the GPU storage equation. The audit below assumes you know the concept and want to measure whether it fits your cluster.

Before any purchase decision, you need data on six audit metrics:

  • GPU idle time
  • Checkpoint write and recovery time
  • East-west traffic
  • Replication scope
  • External-storage demand
  • Time-to-first-token (TTFT)

How Do You Run a GPU Storage Utilization Audit?

A GPU storage utilization audit has three steps. First, inventory what lives on each node’s local NVMe. Second, capture workload-level signals from telemetry you already own. Third, classify every dataset as node-contained or cluster-visible. You record the results in one worksheet row per node, per workload window.

Step 1: Inventory What Lives on Local NVMe

List each NVMe device, its filesystem, mount point, and current contents. Typical contents include container images, dataset caches, per-rank scratch, checkpoint shards, model weights, and KV cache spill. Note who writes each class of data and how long it is retained.

Standard Linux tooling covers this. Use lsblk and df for layout and usage, and nvme-cli for device health logs as defined in the NVM Express specifications. The goal is an honest picture of used versus provisioned space, not a target number.

Step 2: Capture Workload-Level Signals

Each metric has a natural telemetry source:

  1. GPU idle time: node telemetry and GPU monitoring, correlated with scheduler and job logs. This correlation separates a true GPU idle time storage bottleneck from scheduling gaps.
  2. Checkpoint write and recovery time: framework checkpoint logs for save and load events, plus scheduler logs for restart timestamps.
  3. East-west traffic: switch counters on the fabric between compute nodes, split by storage traffic where your fabric allows.
  4. Replication scope: storage monitoring and policy configuration, showing which data has copies and across which failure domains.
  5. External-storage demand: storage monitoring on external arrays, attributed to GPU jobs by client or path.
  6. Time-to-first-token: inference service metrics, especially cold starts and model loads.

Capture distributions over a representative window that includes training runs, checkpoints, and restarts. A single snapshot hides the bursts that matter.

Step 3: Classify Data as Node-Contained or Cluster-Visible

Node-contained data is read only by processes on the same node, and losing it costs only a re-fetch or recompute. Per-rank scratch and local caches of durable datasets fit here.

Cluster-visible data must be readable from other nodes. This class includes checkpoints (restarts land on different nodes), shared training datasets, model weights served by multiple replicas, and RAG corpora. Within this class, flag anything that must survive node loss.

A common mistake is classifying checkpoints as node-contained because they are written locally. That classification holds only until a node fails.

The Per-Node Worksheet: Column Definitions

The worksheet is the core deliverable of this audit. Anyone on your team can rebuild it from the same telemetry and reach the same conclusions.

  • Node ID and failure domain: rack, power feed, and (in cloud) availability zone.
  • NVMe inventory: devices, filesystem, provisioned and used space.
  • Contents by class: node-contained, cluster-visible, or must-survive-node-loss.
  • GPU idle time: observed during I/O-bound phases, with the telemetry source noted (node telemetry and GPU monitoring, scheduler and job logs).
  • Checkpoint write time and checkpoint recovery time: taken from framework checkpoint logs and scheduler logs.
  • East-west traffic: storage-attributable bytes from switch counters.
  • Replication scope: copies and the failure domains they span, from storage monitoring and policy configuration.
  • External-storage demand: load from this node’s jobs on external arrays, from storage monitoring.
  • TTFT: recorded for inference nodes only, from inference service metrics.
  • Notes: manual staging steps, cleanup scripts, and known pain points.

To learn more about how the Hammerspace architecture relates to the data in your worksheet, contact Hammerspace or call +1 (650) 777-8728.

Local NVMe vs Shared Storage: How Do the Three Architectures Compare?

The three architectures trade speed, visibility, and durability differently. Isolated local NVMe is fast but invisible to other nodes. External shared flash is visible and durable but sends every I/O across the network. Policy-managed Tier 0 makes local NVMe cluster-visible while relying on durable tiers for protection.

Scored against the six audit metrics, the tradeoffs look like this:

Audit metricIsolated local NVMeExternal shared flashPolicy-managed Tier 0
GPU idle timeStrong for node-contained data; weak when jobs need data from other nodesDepends on network and array headroomStrong when policy places data local to consuming GPUs
Checkpoint write and recovery timeFast writes; recovery fails if the node is lostDurable from first write; write time depends on networkLocal landing; recovery depends on protection policy completing
East-west trafficLow, but manual staging creates burstsAll storage I/O traverses the fabricReduced for local reads; mirroring and remote reads add traffic
Replication scopeNoneDefined by the arrayFile-granular, set by policy
External-storage demandHigh for anything sharedCarries the full loadLower hot-tier load; durable tier still required
Time-to-first-tokenRequires manual staging of weightsDepends on network during model loadStrong when weights are pre-staged by policy

External shared flash remains the right answer in several situations:

  • Data must be durable at the moment of the first write, with no protection lag.
  • Local NVMe is already full of legitimate node-contained data.
  • Many non-GPU clients depend on the same data.
  • Your audit shows low GPU idle time and ample network headroom.

As the Tier 0 product page makes clear, Tier 0 is a tier within a broader data platform, not a replacement for durable storage. Many clusters end up combining both.

How Should AI Checkpoint Storage Handle Recovery and Failure Domains?

Checkpoints on local NVMe are safe only after policy has made them visible to other nodes and protected them beyond the writing node’s failure domain. Until then, treat every locally landed checkpoint as exposed.

The Four-Stage Checkpoint Lifecycle

  1. Land: shards write to the writing node’s local NVMe. Your checkpoint write time metric measures this stage.
  2. Make visible: the Unified Data Plane presents the shards in a shared namespace, so any node can read them on restart.
  3. Protect: Global Data Services policies define where additional copies live, such as another node in a different failure domain or a durable external tier. Local NVMe is not durable on its own, and protection exists only where a policy creates it.
  4. Drain or tier: Data Orchestration moves older checkpoints to durable external or object storage in the background. Draining frees local capacity for the next save.

A Hypothetical Node-Failure Walkthrough

Imagine a training job spread across several racks. Reason through three failure domains: the node, the rack, and the power feed.

  • Node failure, protection complete: the latest checkpoint’s shards from the failed node were already protected in another rack. The scheduler restarts the rank on a replacement node, which reads the shards through the namespace, so recovery time is essentially read time.
  • Node failure, protection incomplete: the job falls back to the previous protected checkpoint and loses one checkpoint interval of work.
  • Rack or power failure: any copies kept within that rack or power domain disappear along with the primary. Replication scope must span racks and power domains, not just nodes.

Your checkpoint recovery time metric captures protection lag directly. Design policies against the mirroring and tiering options that Hammerspace currently documents, not against assumptions.

Does Local NVMe Reduce Storage Network Bandwidth for GPU Clusters?

Local NVMe reduces some storage traffic, but it does not remove the need to plan storage network bandwidth for GPU clusters. Bandwidth requirements still scale with GPU count, because mirroring, remote reads, ingest, and checkpoint draining all move data between nodes and tiers.

Your audit’s traffic measurements feed network sizing directly. Split east-west traffic from switch counters into three categories:

  • Reads served locally
  • Reads served from other nodes
  • Protection traffic from mirroring

Then measure north-south traffic separately: ingest from data lakes, drains to durable tiers, and the external-storage demand that remains.

Placement policy reduces traffic in specific ways. Pre-staging datasets and weights near the GPUs that consume them cuts repeated remote reads, and scheduling drains outside checkpoint windows smooths bursts.

Policy cannot eliminate traffic that is structurally required:

  • First-touch reads of new data
  • Protection copies across failure domains
  • Cross-node restarts after failures
  • Movement to durable tiers

Protocol choice also shapes the data path. The pNFS flexible file layout, defined in RFC 8435, lets clients read and write directly to data servers, which keeps the metadata server out of the data path. That parallel data path is why pNFS for AI workloads avoids funneling I/O through a single controller. For more on this architecture in large clusters, see the Hammerspace overview of GPU storage for AI and HPC infrastructure.

How Does the Audit Apply to Inference, RAG, and Agentic AI?

The same node-contained versus cluster-visible classification applies to every workload beyond training. What changes is which metrics dominate: TTFT and external-storage demand matter more than checkpoint recovery.

Inference

Model weights read by many replicas are cluster-visible, read-heavy, and reproducible from a durable source. Cold starts show up directly in TTFT. Hammerspace positions its platform for both training and inference, and file-granular Service-Level Objectives let policy pre-stage files near GPUs before replicas need them.

KV cache spill tied to a single replica is node-contained. If your serving stack shares cache across replicas, classify it as cluster-visible and audit its traffic accordingly.

RAG Pre-Staging

RAG corpora and embeddings are cluster-visible, and they often span both files and objects. Hammerspace unifies file and object access in one namespace, so policy can place embeddings and corpora close to retrieval services. Index refreshes generate north-south traffic, so record them in your external-storage demand column.

Agentic AI

Agentic workloads create many small intermediate artifacts. One agent writes an artifact that another agent reads, often on a different node, which gives the pattern cluster-visible semantics even when individual files are short-lived.

Treat agentic AI as an access-semantics question in your audit. Confirm with Hammerspace which configurations are currently documented for agentic workloads before you design around them.

Can Cloud GPU Instance Local NVMe Be Used as Shared Storage?

Cloud GPU instance local NVMe works as shared storage only when software exposes it into a shared namespace and policy protects it to durable storage. Instance-local NVMe is typically ephemeral and per-instance, so its contents disappear when the instance stops or terminates. That makes the cloud the place where this audit matters most.

Ephemerality sharpens both core questions. First, the stranded-capacity question: the drives come with the instance whether you use them or not, so leaving them idle while provisioning separate shared storage deserves scrutiny. Second, the protection question: any cluster-visible data landing on ephemeral drives must be protected to durable cloud storage by policy, with no exceptions.

Hammerspace materials describe Tier 0 activating local NVMe in cloud instances as well as on-premises servers. See how Tier 0 turns on AI-ready infrastructure you already own, and confirm current support for your specific cloud environment with Hammerspace.

Adjust the worksheet for cloud:

  • Add instance and availability zone to the failure-domain column.
  • Pull instance lifecycle events from autoscaler and scheduler logs, since spot interruptions and scale-downs behave like node failures.
  • Track north-south traffic between cloud and on-premises sites separately, because egress shapes placement policy as much as performance does.

Why Do Open Standards Matter for Tier 0 Storage?

Open standards mean Tier 0 storage does not require a proprietary client on every GPU node. Nodes access data through the native Linux NFS client using NFSv4.2 and pNFS, or through SMB and S3. That removes a layer of kernel modules, version pinning, and upgrade coordination from your GPU fleet.

pNFS was introduced in NFSv4.1, now specified in RFC 8881. NFSv4.2, specified in RFC 7862, adds several features relevant to AI data pipelines:

  • Server-side copy
  • Sparse file support
  • Space reservation
  • Application I/O hints

The mainline Linux kernel NFS client implements server-side copy, sparse file operations, and space reservation. Check your kernel version for the specific operations your pipelines rely on.

Proprietary client-based stacks work differently. They typically require a vendor module on each node, and that module must stay compatible with kernel updates and GPU driver releases. Your data access path is also tied to one vendor’s roadmap.

A standards-based design keeps your audit metrics portable. You can change hardware or add sites, and external-storage demand shifts through policy rather than migration. Metadata assimilation lets existing storage stay in place while Tier 0 joins the same namespace.

The protocol lineage is also a matter of record, offered here as context rather than as a performance claim. Hammerspace’s team includes people who built and shaped Linux NFS: Trond Myklebust, long-time maintainer of the Linux NFS client; Tom Haynes, author of the NFSv4.2 specification; and Brian Pawlowski, co-author of the NFSv3 specification. CEO David Flynn leads the company.

How Do You Turn GPU Storage Audit Results Into an Architecture Decision?

Quantify your own cost-model variables from the worksheet, then match the signal patterns you observe to an architecture. The decision rests entirely on your data, not on vendor benchmarks.

Cost-Model Variables to Quantify

  • GPU hours lost to storage stalls, derived from GPU idle time
  • Work lost per failure, from checkpoint interval plus recovery time
  • External flash capacity and throughput growth attributed to GPU jobs
  • Network upgrades driven by measured east-west and north-south traffic
  • Operational effort spent on manual staging and cleanup
  • Rack space and power consumed by external arrays

Signal Patterns

These patterns point toward Tier 0:

  • Local NVMe is underused.
  • GPU idle time correlates with remote reads.
  • Cluster-visible data is read-heavy.
  • Checkpoint write time stalls training.

These patterns point toward external shared flash:

  • Local NVMe is already full of node-contained data.
  • Data must be durable at the moment of the first write.
  • Many non-GPU clients depend on the same data.
  • GPU idle time is already low.

These patterns point toward a combination:

  • Hot, cluster-visible data benefits from locality.
  • Durability and long-term retention still require external or object tiers.

For many teams, the combination wins: Tier 0 as the hot tier, durable storage underneath, and policy moving data between them.

Your audit shows whether GPU server local NVMe in your cluster is stranded capacity or a tier waiting to be activated. It also pinpoints which workloads, checkpoints, and network paths are involved. To learn how the Hammerspace High Performance Data Platform for AI Anywhere, with its Unified Data Plane, Global Data Services, Data Orchestration, and Tier 0, applies to the patterns in your results, contact Hammerspace or call +1 (650) 777-8728.

Frequently Asked Questions

What is stranded NVMe capacity in GPU servers?

Stranded NVMe capacity is fast local flash inside a GPU server that only its own node can see and use. The drives are installed and often underused, but no other node can read from them. As a result, shared data such as checkpoints, datasets, and model weights goes to external storage instead. The outcome is a utilization gap: provisioned flash sits idle while teams keep expanding external arrays.

Can local NVMe in GPU servers be shared across nodes without a proprietary client?

Yes, if software exposes the drives through standard protocols. Hammerspace Tier 0 presents local NVMe in a shared namespace. GPU nodes reach it through the native Linux NFS client using NFSv4.2 and pNFS, or through SMB and S3. Because no vendor kernel module is required, you avoid compatibility work every time kernels or GPU drivers change. Verify that your kernel version supports the specific NFSv4.2 operations your pipelines depend on.

Should AI training checkpoints be written to local NVMe or shared storage?

The answer depends on how quickly checkpoints can be made cluster-visible and protected. Writing locally can shorten the time training stalls during a save. However, a checkpoint that exists only on the writing node is lost if that node fails, and restarts often land on different nodes. With policy-managed Tier 0, checkpoints land locally, appear in the shared namespace, and are then copied to another failure domain or a durable tier. If your requirements demand durability at the moment of first write, external shared storage is the safer choice.

How is data on Tier 0 protected if a GPU node fails?

Protection comes from policy, not from the local drive itself. Global Data Services policies define where additional copies of files live, such as a node in a separate rack or power domain, or a durable external or object tier. If a node fails after protection completes, a replacement node reads the data through the shared namespace. If protection has not finished, recovery falls back to the last protected copy. For this reason, replication scope should span racks and power feeds rather than just individual nodes.

Does using local NVMe reduce storage network bandwidth requirements for a GPU cluster?

Local NVMe reduces some traffic, but you still need to plan network bandwidth. Reads served from a GPU’s own node stay off the fabric, and pre-staging data near the GPUs that consume it cuts repeated remote reads. Some traffic remains unavoidable, including mirroring across failure domains, first-touch reads, cross-node restarts, ingest, and drains to durable tiers. Split your measured traffic into local reads, remote reads, protection traffic, and north-south flows, then size the network from those numbers.

Can Tier 0 use local NVMe on cloud GPU instances?

Hammerspace materials describe Tier 0 activating local NVMe on cloud instances as well as on-premises servers, but you should confirm support for your specific cloud environment. Cloud instance NVMe is typically ephemeral, so its contents vanish when an instance stops or terminates. Any cluster-visible data landing on those drives must be protected to durable cloud storage by policy. Plan for spot interruptions and scale-down events the same way you plan for node failures, and include availability zones in your failure-domain design.

Is Tier 0 a replacement for external storage or a complement to it?

Tier 0 is a complement to external storage. It acts as a hot, cluster-visible tier built on local NVMe. Durable external or object storage still handles protection, long-term retention, and any data that must be durable at first write. Data Orchestration moves data between tiers in the background, and metadata assimilation lets existing storage stay in place within the same namespace, so many clusters run both together.

Data Orchestration For Dummies

  • Unlock and monetize your data
  • Achieve a unified global data platform
  • Liberate from data silos
Free Download

Share

Make AI Anywhere, A Reality!

See how Hammerspace can unify all your data, accelerate your AI workloads, and deliver results faster.
Get Started

Related Blog Posts