At SC24, Hammerspace is set to demonstrate how its Global Data Platform is transforming unstructured data management for AI and GPU computing environments. Fresh from powering Meta's AI Research SuperCluster with 24,000 GPUs at 12.5TB per second, the company will highlight its standards-based parallel file system architecture that eliminates data silos across edge devices, data centers, and clouds.

Visitors to Booth #4723 can meet comic book legend Steve Gordon, who will be signing the latest volume of the company's "Data Dynamo" series, while technical experts showcase solutions that have recently achieved notable results in the MLPerf Storage Benchmark tests. The company's biggest presence yet at SC24 includes a cocktail reception, technology partner demonstrations, and participation in the Petaflop party at the College Football Hall of Fame.

Learn more in this VMblog Q&A with Molly Presley, SVP of Global Marketing, at Hammerspace.

VMblog: If you were giving an SC24 attendee a quick overview of the company, what would you say?  How would you describe the company?

Molly Presley:  Hammerspace radically changes how unstructured data is used and preserved. Our Global Data Platform unifies unstructured data across edge devices, data centers, and clouds. It provides extreme parallel performance for AI, GPUs, and high-speed data analytics, and orchestrates data to any location and any storage system. This eliminates data silos and makes data an instantly accessible resource for users, applications, and compute clusters, no matter where they are located.

The company was founded with a mission to make data an instantly accessible globally shared resource, constantly in motion, no longer trapped within storage silos. Our founder and CEO, David Flynn, was previously co-founder and CEO of Fusion.io, a pioneer in flash storage technology.

Hammerspace has deep technical roots in the Linux, NFS, and open-source communities, and has made extensive contributions to the Linux NFS kernel, including work that has pioneered a standards-based parallel file system architecture now deployed in some of the largest GPU computing environments in the world.

VMblog:  Your company is sponsoring this year's SC24 event.  Can you talk about what that sponsorship looks like?

Presley:  SC24 will be our biggest event of the year - biggest booth, biggest team and biggest presence! We are hosting a cocktail party in our booth, are sponsoring various parties during the event, and are launching the second volume of our Data Dynamo series comic book, illustrated by famed comic artist Steve Gordon, who will be signing the comic books in our booth!

VMblog:  Thinking about your company's technologies, what are the types of problems you solve for an SC24 attendee?

Presley:  Hammerspace helps organizations to:

• Eliminate data silos and unify all of their unstructured data with a single global namespace that spans sites, clouds, and storage
• Meet the extreme storage performance requirements of AI, HPC, and GPU computing with a standards-based parallel file system architecture
• Automate the flow of data between local and remote compute resources with metadata-driven data orchestration services

VMblog:  While thinking about your company's technology and solution offerings, can you give readers a few examples of how your offerings are unique?  What are your differentiators?  What sets you apart from the competition?

Presley:  Our customers running compute-intensive workloads select Hammerspace for five main reasons.

Reason # 1: Performance that Scales Linearly, and Can Feed Thousands of GPUs Concurrently

Hammerspace is based on a parallel file system architecture which is presented as a NAS for applications and users to connect over standard NFS. The software leverages Parallel NFS and Flex Files to create a direct data path between Linux clients (in this case the GPU servers) and the underlying storage nodes (which can be off-the-shelf or OCP commodity servers).

Reason # 2: Everything is Standards-Based

The Hammerspace Hyperscale NAS architecture is unique because it provides HPC-class performance (like HPC file systems) using standards-based NFS 4.2; a standard client that is part of the Linux kernel. In other words, no proprietary file system clients, no complex NFS client customizations. Existing client servers on any modern build of Linux already can serve as a parallel file system client and achieve both massive and efficient data path performance that is not possible with scale-out NAS systems.

Hammerspace achieves this by being the first and only file system that uses Parallel NFS and Flex Files - optional capabilities of the NFS 4.2 specification that were engineered by Hammerspace and made part of the NFS 4.2 specification in 2018. Hammerspace has been and will continue to be a major contributor to the Linux community in the NFS protocol.

Reason # 3: Efficient Architecture That Can Save $Millions

Rather than purchase expensive custom or purpose-built hardware from storage vendors, customers can build Hammerspace storage clusters using commodity off-the-shelf servers - and these can be heterogeneous, meaning not all of the servers need to have identical specifications.

Typically these clusters would be built with NVMe storage servers to get the lowest latency and highest performance per rack unit, but Hammerspace storage clusters can be built with any combination of NVMe, SSD, and HDD storage.

Reason # 4: Hammerspace Makes It Easy to Assemble and Consolidate Large Datasets for AI Training

Hammerspace offers a unique capability called Data-in-Place Assimilation. This feature allows Hammerspace to import metadata from existing file storage systems, while leaving the data itself in place, so that the files stored on those storage volumes are immediately visible to users and applications. Data orchestration can move data to new storage systems or alternate compute sites automatically using Hammerspace Objectives (our word for data policies), and this happens transparently and non-disruptively to users. These policies can also be set on a file-granular basis, so move only the files you need.

Reason # 5: GPUs Not Co-Located with Your Training Data? No Problem!

Right now, just procuring GPUs is a challenge. And for many companies in the early stages of their AI journey, they may not be willing to make a big capital investment in a dedicated GPU infrastructure. So if your GPU infrastructure is not co-located with your training data - no problem!

Hammerspace data orchestration for compute can move directories, and even individual files, between sites and between clouds. This provides a very fast and very efficient data transfer, and even allows for cloud bursting if you want to access compute resources (including GPU resources) with various cloud providers.

This Global Data Platform eliminates data silos and makes data an instantly accessible resource for users, applications, and compute clusters, no matter where they are located.

VMblog:  How can attendees of the event find you?  What do you have planned at your booth this year?  What type of things will attendees be able to do at your booth? 

Presley:  Hammerspace will have a significant presence at SC24 - our top executives will be there along with technical experts to brief attendees on our solutions for HPC and AI storage, and hybrid-cloud data orchestration. Additionally, we'll have several technology partners exhibiting in our booth (#4723), including QStar, Parallel Works, and Starfish.

I'll be hosting and conducting live podcast interviews with industry thought leaders in our booth for the Data Unchained podcast.

Plus, grab the latest issue of Hammerspace's comic book "The Rise of Entropy" to uncover the latest adventures of our hero Data Dynamo!  Read about Dave, a tech employee turned superhero, as he battles the evil Entropy to free the world's data! Illustrated by comic book legend and artist Steve Gordon, he will be autographing your very own copy in our booth throughout the show! We will also have a giant space-themed plinko board where participants can win fun prizes!

Hammerspace will also be hosting a cocktail reception during the Grand Opening Gala Reception in our booth on Monday, Nov. 18 (7 p.m.), and sponsoring the Petaflop party on Tuesday, Nov. 19 at the College Football Hall of Fame, where we will raffle of 11 x 17 original artwork from Steve.

If you're interested in an architecture that delivers HPC performance without the complexity of a proprietary file system client, can reduce latency by 80% or more by leveraging local NVMe storage, and makes it easy to orchestrate and run HPC and AI workloads in the cloud, stop by Booth #4723!

VMblog:  Have you sponsored SC24 in the past?  If so, what is it about this show that keeps you coming back as a sponsor?

Presley:  We have sponsored SC24 for the past few years and keep coming back because it provides a direct connection to our future and current client base. 

VMblog:  What major industry challenges or technological bottlenecks do you see your solutions addressing in the high-performance computing (HPC) landscape, particularly in relation to emerging AI/ML workloads?

Presley:  One of the standout examples of Hammerspace's capabilities is its use by Meta, which operates one of the world's largest hyperscale AI pipelines. Meta relies on Hammerspace to orchestrate data between 1,000 existing NVMe storage servers to feed the AI Research SuperCluster of 24,000 GPUs at a staggering 12.5TB per second. This solution allows Meta to avoid the costly and time-consuming process of moving data between storage systems, while still taking full advantage of its existing commodity storage and compute resources.

Additionally, in the recently published MLPerf Storage Benchmark results which simulates AI workloads, Hammerspace was the only vendor to run the tests entirely on cloud-based compute resources, matching or beating the results of dedicated on-prem HPC hardware configurations.

Hammerspace was the only enterprise-class NAS solution that submitted results for the benchmark, leveraging its standards-based Hyperscale NAS architecture. This architecture utilizes the extreme parallel file system scalability pNFSv4.2, but is standards-based, without the need for client software, special networking, and with support for existing storage from any vendor.

For enterprises looking to scale AI workloads without overhauling their existing storage infrastructure, Hammerspace offers a way to automate data orchestration across multi-vendor siloed environments. Its ability to span both on-premises and cloud storage in a unified global namespace means that organizations can take advantage of cloud-based or remote GPU clusters without having to migrate all their data into net new repositories on premises or in the cloud.

VMblog:  The convergence of HPC and AI continues to reshape the industry. How are your solutions enabling organizations to bridge traditional HPC workloads with new AI/ML demands, and what unique capabilities will you be demonstrating at the event?

Presley:  The Hammerspace Global Data Platform enables seamless, automated data orchestration between on-premises storage systems and cloud-based computing environments, including the ability to do so without wholesale data migrations.

One of the key innovations of Hammerspace is its ability to decouple the high-performance file system from the underlying storage infrastructure to enable this orchestration to become seamless in the background. This allows data to remain in place-whether on-premises or in the cloud-while still being accessible at high performance across distributed environments.

By bridging incompatible storage silos, Hammerspace eliminates the need to move massive datasets into the cloud just to process AI workloads. Instead, the system automates the orchestration of data at a file-granular level to remote compute resources as needed, allowing AI models to access the data directly, wherever it resides.

This capability ensures that high-performance AI applications-such as deep learning model training, inferencing, and analytics-can be fed with the right data at the right time, whether that data is stored on-premises, in the cloud, or across multiple geographic locations. And that this can be done automatically, and without the need to migrate whole data sets.

VMblog:  Data movement and storage continue to be critical challenges in HPC. What innovations is your company bringing to market to address these bottlenecks, and what performance improvements can customers expect?

Presley:  Unstructured data orchestration is the process of taking siloed file and object data from multiple storage locations, consolidating that data into a global namespace that spans sites and clouds, and then automating the flow of data within that Namespace based on business needs.

Hammerspace's Data Orchestration solves key challenges for IT infrastructure teams like simplifying data governance by eliminating data copy sprawl, freeing up IT resources by automating manual processes, and enabling these teams to take full advantage of elastic cloud compute and storage resources.  

Files are distributed across storage systems, both on-premises and across different clouds and cloud regions, so data can be brought to the compute resources, applications, and users that need it, when they need it.

Examples of the way our customers are using data orchestration today include:

• Moving data to cloud compute resources for rendering or burst computing
• Intelligently tiering data from high-cost performant storage to low-cost capacity storage
• Automating manual data management tasks
• Speeding up the process of making local data available to employees at remote offices
• Automating the movement of data between cloud regions
• Automating the movement of data between distributed edge sites and core data centers
  

VMblog:  Looking ahead to 2025 and beyond, what major trends do you see shaping the future of supercomputing, and how is your company positioning itself to lead in these areas?

Presley:  

(1)    GPU-Centric Data Orchestration Becomes Top Priority - As we head into 2025, one of the challenges in AI and machine learning (ML) architectures continues to be the efficient movement of data to and between GPUs, particularly remote GPUs. This is becoming a critical architectural concern as companies scale their AI/ML workloads across distributed systems. Traditional data orchestration solutions, while valuable, are increasingly inadequate for the demands of GPU-accelerated computing.

The bottleneck isn't just about managing data flow-it's specifically about optimizing the transport of data to GPUs in real-time, often over networks, to support high-performance      computing (HPC) and advanced AI models. As a result, the industry will see a surge in innovation around GPU-centric data orchestration solutions, like Hammerspace. These new systems will focus on minimizing latency, maximizing bandwidth, and ensuring that data can seamlessly move across local and remote GPUs.

(2)    Breaking Down Data Silos Will Become a Central Focus for AI and Data Architects - In 2025, breaking down data silos will emerge as a critical architectural concern for data engineers and AI architects. The ability to aggregate and unify disparate data sets across organizations will be essential for driving advanced analytics, AI, and machine learning initiatives. As the volume and diversity of data sources continue to grow, overcoming these silos will be crucial for enabling the holistic insights and decision-making that modern AI systems demand.

The focus will shift from the infrastructure toward the seamless integration of data across various platforms, teams, and geographies. The goal will be to create an ecosystem where data is easily accessible, shareable, and actionable across all domains. Expect to see new tools and frameworks aimed at simplifying data integration and fostering greater collaboration across traditionally siloed environments.

(3)    Enterprise HPC Must Align with Standardized Technologies for Unstructured Data Processing - By 2025, medium to large enterprises will face a pivotal challenge: integrating high-performance computing (HPC) for unstructured data processing while adhering to enterprise standards. As organizations increasingly rely on AI and data analytics to gain a competitive edge, the need to process vast amounts of unstructured data-like text, images, and video-will be unavoidable. However, enterprises have long struggled to adopt HPC at scale due to the complexities of reconciling specialized HPC technologies with enterprise requirements for security, compliance, and operational standards.

The solution lies in the development of HPC technologies designed to work within enterprise-standard environments. In 2025, we expect to see the rise of enterprise-ready HPC solutions that seamlessly integrate with standard clients, operating systems, networks, and security frameworks. This convergence will enable organizations to finally leverage HPC for large-scale unstructured data processing without compromising on enterprise security, compliance, or performance standards.

VMblog:  Are you giving away any prizes at your booth or participating in any prize giveaways?

Presley:  We are! We are raffling off original artwork from famed comic artist Steve Gordon. We are giving away autographed copies of both Volume 1 and Volume 2 in our Data Dynamo comic series. Participants in our Plinko game can win a Data Dynamo t-shirt, a Data Dynamo inflatable doll, and cosmic stress balls.

VMblog:  Is your company sponsoring any type of party or get together during the event that you'd like to notify attendees about?

Presley:  Yes! We are an exabyte sponsor of the Petaflop party, held on Tuesday evening at the College Football Hall of Fame, located across the street from the convention center. Stop by our booth to pick up an invitation!

We are also hosting a cocktail reception during the Grand Opening Gala Reception in our booth on Monday, Nov. 18 (7 p.m.).

##