Hammerspace and
Parallel Works
unveiled a unified solution for compute and data orchestration. This
groundbreaking solution enables IT infrastructure teams, research
computing teams and AI teams to orchestrate compute, storage and data
workloads seamlessly across sites and clouds, making it easy to scale
cloud compute resources for burst capacity.
"Innovators need the agility to move complete workloads across
different compute clusters - adding flexibility to use CPUs for data
preparation, GPUs for processing, or accessing remote clusters for extra
cycles or cost-efficient cloud regions. Seamlessly orchestrating both
compute and data across locations and cloud regions is a game-changer,"
said David Flynn, Founder and CEO of Hammerspace. "Our unified solution
with Parallel Works simplifies operations, reduces overhead and
accelerates innovation by ensuring data is always where it needs to be,
with minimal latency."
"This partnership marks a major advancement in managing compute
resources and data across hybrid and multi-cloud environments," said
Matthew Shaxted, CEO of Parallel Works. "By integrating our expertise in
HPC and AI workload orchestration with Hammerspace's parallel file
system, global namespace and data objectives, we're enabling application
portability and achieving agile task placement. This approach empowers
organizations to efficiently allocate resources where needed, driving
performance, innovation, and simplified hybrid infrastructure
management."
AI and HPC workloads demand significant compute resources and
low-latency access to large datasets. However, both data and compute
resources are often distributed across different locations, making it
challenging to process data efficiently. Leveraging alternate data
centers or cloud infrastructure provides a scalable, elastic option for
burst compute capacity, enabling on-demand scaling. Yet, historically,
orchestrating workloads across multiple, geographically dispersed
clusters - especially when data is remote from the compute - has been
highly complex. This complexity arises from the need to relocate the
application while managing data transfer, network latency and
synchronization, leading to error-prone and time-consuming setups that
most teams have been unwilling to take on the burden of managing.
Hammerspace and Parallel Works have co-developed a unified,
integrated solution for compute and data orchestration that automates
the provisioning and management of local and cloud compute resources and
orchestrates the flow of data to those compute resources.
In this solution, Parallel Works ACTIVATE acts as the unified control plane
that creates, provisions, and manages the compute resources located in
multiple data centers and clouds. Parallel Works also automates
workflows, such as spinning up Slurm instances to manage job scheduling
and the associated HPC nodes.
Hammerspace acts as the data platform that provides
high-performance file and object storage, creates a Global Namespace
that spans sites and clouds, and automates the flow of data within that
namespace with Data Orchestration services. Hammerspace delivers the
data to the HPC servers regardless of where they reside, with
orchestration policies set at the share, directory, and even individual
file levels.
This integrated solution enables administrators to create, configure,
and manage storage clusters from the Parallel Works ACTIVE Control
Plane console.
By making it easier to use elastic cloud resources for burst
computing capacity, organizations can "right size" their infrastructure
with the ease of use of a fully managed cloud service but at the cost
point of standard cloud infrastructure services. This approach can
deliver significant cloud cost savings, as shown in the chart below that
compares the annual costs of Hammerspace on standard cloud virtual
machines to Managed Lustre and shows the savings potential of the
Hammerspace standards-based architecture.