Leveraging Storj Distributed Cloud with Hammerspace Global File System

Overview

This technical reference illustrates how to successfully move existing data from on-premises storage to Storj Cloud. Hammerspace provides a Global Namespace and Global File System that makes data migration completely non-disruptive and transparent to applications and end-users. This is vitally important for the modern and geographically distributed enterprise. Moving data into Cloud with Hammerspace is distinctly different from disruptive legacy data migration solutions that often lead to additional silos and lost productivity. Moving data between volumes, storage systems, data centers, and clouds is a requirement for the modern enterprise to operate efficiently.

Architecture

The topology diagram below illustrates a simple standalone Hammerspace deployment with on-premises storage, end-users, and Storj Distributed Cloud. Hammerspace provides multiprotocol access to applications and end-users through SMB and NFS. A CSI driver is also provided for Kubernetes Storage.

Configuration

This section will walk through each of the steps required to add storage to Hammerspace. We will cover the prerequisites, how to add a storage system and turn that into a Hammerspace volume. We will, then, show how to make data available through a multiprotocol interface with NFS and SMB. Finally, we will also demonstrate how to migrate data non-disruptively from on-premises storage to Storj Cloud.

Prerequisites

Anvil metadata node installed.
DSX data node installed.
One or more Hammerspace shares.
On-Premises storage system added to Hammerspace.
Storj account credentials.
A Storj bucket available for use.

Adding Storj Distributed Cloud Storage to Hammerspace

Log into the Hammerspace Management GUI by pointing your web browser to the IP address of the Anvil metadata node. Hammerspace will automatically redirect to HTTPS and TCP port 8443 if not entered, as shown below.

https://<ip address>:8443

Log into the Hammerspace Admin GUI by providing the username and password.
Username: admin
Password: set during initial deployment of Hammerspace

Select the highlighted “Infrastructure” icon in the left navigation pane.
Select the “Storage Systems” tab.
Click the “Add Storage System” button to open the Add Storage System configuration window.

Fill out the form fields in the configuration window.
- Provide a meaningful name
- Type: select Generic S3 from the dropdown menu.
- Access Key: obtain from your Storj account.
- Secret Key: obtain from your Storj account.
- Endpoint: obtain from your Storj account.
- Trust self-signed certificates: optional
- Use virtual host bucket signing: optional
- Use V4 auth signing: Select the checkbox.
- Proxy Host: optional
- Proxy Host: optional
Click “Add Storage System” to finalize the configuration and kick off the process.

Create a Volume

After adding a new storage system to Hammerspace we need to create a volume to make data shareable through the multiprotocol front-end interface.

Navigate to “Infrastructure” from the left navigation pane.
Select the “Storage Systems” tab.
Click the “+ Volume” button in the column listing Storj.
Follow the steps in the wizard that opens.

Select the checkbox(es) for the Storj bucket(s) to add to Hammerspace. In this example, we are adding a single bucket, but it is just as simple to add multiple buckets at once if desired.
Click “Next Step >”.

IMPORTANT: Do not check the “Shared Volume” checkbox. This must be empty unless creating a Hammerspace Global File System. This is a separate topic covered in other Hammerspace documentation. This can be done at any time before or after adding storage to Hammerspace. Please contact Hammerspace if you are configuring a Global File System.

Select a desired capacity if you wish to limit the capacity or choose “Auto” for unlimited capacity.
Click “Next Step >” and the new volume will be created in a moment. It should look something like the image below.

Finally, review the input parameters and click “Add Volume”.
The new volume is now ready for use.

Create a Share

This optional step illustrates how to create a multiprotocol share with the SMB and NFS. We could just as easily leverage an existing share for our example. You can also use a previously created share for this purpose.

From the left navigation pane, select the “Shares” icon.
Click the “Create Share” button.

The “Create Share” configuration window will open
From the “Share Details” tab fill out the following information:
- Name: Provide a name for the SMB share.
- NFS Export Path: automatically filled out (from the “Name” form field above).
- Max Share Size: Set the desired maximum size for the share.
- Alert Threshold: Set a percentage of capacity trigger for alerting (soft quota).

Please note that there are additional options for NFS and SMB as shown below.

Select the “NFS” tab to view or edit additional configuration options. These are standard NFS configuration options, including:
- Export Subnet
- Permissions
- Root Squash
- Secure Port
Click “Update” if making changes.
Next, let’s look at the additional SMB configuration options.

This tab allows us to configure:
- SMB Aliasing. NOTE: The name must be unique across all shares.
- SMB Alias Path (relative to the root of the share). NOTE: Must begin with a forward slash (“/”).
- Whether or not to make the Share browsable (checked by default).
Click “Update” if making changes (optional).
Select “Close” to finalize the share configuration.

NOTE: We will look closer at Objectives in the next step. Snapshot Schedules and Global File System configurations are treated in detail in additional documentation which can be obtained by contacting Hammerspace.

Non-Disruptive Cloud Migration

Hammerspace has a unique ability to move data between volumes, storage systems, sites, and clouds non-disruptively. This is completely different from legacy data migration that requires service windows and disruptive downtime. Hammerspace Data Mobility is live data movement with zero interruptions to applications, end-users, and services. Uptime remains 100% while moving data from point-to-point or point-to-multipoint with Hammerspace Live Data Mobility. Let’s return to our Share configuration in the Hammerspace Admin GUI.

Select the “Objectives” tab.
Click “Add Objective”.
Select “place-on-My-Storj” from the Objective dropdown menu.
Click “Add” to apply the Objective.
Click “Close”.
Next, we will validate that data is moving in alignment with the Objective.

The Objective will move data non-disruptively from on-premises storage to Storj Cloud. The name “My-Storj” was chosen in the section “Adding Storj to Hammerspace”. This name will vary depending on your configuration choices.

Observability

The final step is to validate the data movement to Storj. To accomplish that we turn to something we call alignment. This can be viewed in multiple ways. Let’s turn our attention to the Mobility area of the Admin GUI.

Select the “Mobility” icon from the navigation pane on the left. The Chord provides a high-level graphical representation of data mobility. This example uses a simple view of moving data from one volume to another. It is important to note that this is not restricted to one-to-one movement. It could just as easily be one-to-many, many-to-one, or many-to-many data movement with Hammerspace Objectives.
Click the “View as List” button in the upper right corner for a more detailed matrix representation of the data movement.

From this view we can observe the following:

File(s) and share(s) affected by the data mobility Objective.
Source and destination volumes.
Status.
Why the data is being moved, i.e. the desired end-state implemented by the Objective.
Throughput.
Start time and completion time for each file transfer.

Summary

Hammerspace provides a Global Namespace that aggregates and consolidates disparate storage systems into a single cohesive and coherent logical unit. Moving data from on-premises storage to cloud with Hammerspace is simple, seamless and, most importantly, non-disruptive. Why endure downtime, lengthy maintenance windows, disruptive migration, and lost productivity when it is not necessary?

We have introduced the concept of Hammerspace Objectives and demonstrated their power and ease of use in moving data from on-premises storage to cloud. Data placement and availability have been limited by underlying storage infrastructure for decades. Why settle for something that chains data to infrastructure?

Hammerspace unchains data from underlying infrastructure. Moving data across volumes, storage systems, data centers, and clouds is trivial in a Hammerspace solution. It is our focus to make data available, durable, and secure regardless of underlying infrastructure. This is modern data management at work! Why settle for anything less?