ScaleFlux, FarmGPU, and Lightbits Labs announced the public debut of a collaborative architecture designed to solve one of AI inference’s most persistent challenges: the memory and I/O constraints created by long-context workloads.
At NVIDIA GTC San Jose, in March, the companies will debut an implementation that
brings together ScaleFlux high-performance NVMe, FarmGPU’s managed inference
environment, and Lightbits LightInferra software to solve how KV-cache data can be
persisted, reused, and streamed more efficiently across inference sessions to reduce GPU
stalls caused by repeated context recomputation and open the door to more predictable,
scalable performance and infrastructure efficiency.
“We’re transforming inference memory from a reactive cache into an intelligent, streamed
data layer,” said Arthur Rassmuson, Director of AI Architecture at Lightbits Labs. “By
prefetching only the data that matters and delivering it to GPUs over high-speed RDMA
before it’s needed, we eliminate the stalls that traditionally limit long-context
performance. The result is lower Time-to-First-Token (TTFT), more stable throughput
under real-world load, and significantly higher effective GPU utilization. For enterprises,
that means serving larger models and longer conversations at lower infrastructure cost —
and for end users, it means faster, smoother, more responsive AI experiences.”
“Fast networked storage from Lightbits unlocks a lot of new use cases for long context
inference,” said Jonmichael Hands, Chief Executive Officer at FarmGPU. “By pairing our
managed service with Lightbit’s high-performance storage running on ScaleFlux NVMe,
we are able to lower time to first token and increase utilization on GPUs, drastically
lowering the TCO for inference.”
Key areas under exploration include:
● Higher GPU Utilization and Inference Throughput: Extending and sharing the KV
cache beyond limited GPU memory, enabling the same GPUs to serve up to 3×
more inference requests by eliminating redundant computation.
● Reduced Latency and Increased Stability: Lowering TTFT and Time Per Output
Token (TPOT) by retrieving attention states from storage instead of recomputing
them to mitigate inference stalls as context windows expand.
● AI-Native Security and Isolation: Providing end-to-end security, including
encryption for KV cache blocks, tenant isolation, and integration with Key
Management Systems (KMS) and Trusted Platform Modules (TPM) for shared
inference environments.
“As members of the NVIDIA Magnum IO GPU Direct Network, we see this as an
opportunity to collaborate openly with the ecosystem,” said Keith McKay, Senior Director
of Solutions Architecture and Technical Partnerships at ScaleFlux. “What we’re showing at
GTC is an early look at how smarter data placement and persistent attention state
management could help inference systems stay responsive as context windows grow. This
is very much a collaboration we want to shape alongside real operators.”
This announcement marks the beginning of a design-partner-driven effort, with the
companies actively seeking feedback from AI infrastructure teams, platform builders, and
service providers running large-scale or long-context inference workloads.
