By Chris Sharp, Chief Technology Officer, Digital Realty
Ask most people what AI infrastructure looks like and they’ll describe a room full of GPUs. Fair enough: accelerators are the scarce, expensive part, and every conversation about AI capacity seems to start with how many you can get and how soon.
But a GPU doesn’t answer the questions that actually determine whether AI works in production. It won’t tell you where the workload should run, how it will reach the data it depends on, how it will serve the users waiting on a response, or whether the building it sits in can keep it powered and cooled as it grows. Compute is one input into a much larger design problem, and as more organizations continue running it as part of the business, the rest of that problem is where the hard decisions sit.
We’ve seen this before. The internet was the first great exchange, where traffic and content met. Cloud the second, where applications and data met. AI is the third, with different physics. The first two could largely be solved with fabric, meaning connectivity and interconnection. AI cannot; it needs fabric and power together, in the same place, at a profound scale.
Inference changes where infrastructure belongs
Training gets most of the attention because it is where the density and the headline numbers live. It is also what drives the hardest facility requirements: rack densities heading toward a megawatt per rack, numbers that would have been unthinkable five years ago, liquid cooling as a baseline rather than an exception, and power draw that makes site selection a utility conversation first and a real estate conversation second. None of that goes away as AI matures.
Inference adds a placement problem. A training run can sit in one large campus for weeks. Inference serves repeated live interactions and business processes with a user or a system waiting on the other end. That makes response time, data access and operating conditions central in a way they were not for training alone.
Agentic AI raises the stakes further. Take something simple: an assistant that checks inventory, confirms a price and schedules a delivery. Now, that’s no longer one call to a model, but rather a chain of round trips between the model, the data it needs and the systems it acts on. Each step waiting on the one before it. Traffic stops being a one-way request and response and becomes a running, bi-directional conversation between machines. Every leg of that conversation adds distance, and distance adds delay — so the more agents talk back and forth, the more it matters that the model, the data and the systems sit close together.
No two AI workloads are alike. A customer-facing assistant, a quality system on a factory floor and an internal research tool have very different needs for latency, data proximity, scale and control. Some belong in a large centralized campus where scale is cheapest. Others belong in a metro location next to enterprise data and the cloud on-ramps that feed them. A few belong closer still to the operation they support. Each workload must sit in the environment that fits it, and those placement decisions have consequences. They determine how far data has to move, which services a workload can reach, and how an organization manages data residency and controls across jurisdictions. Geography becomes a critical part of the AI architecture.
Connectivity is necessary, but it is not the system
When placement becomes the question, the instinct is to reach for the network. Connect everything well enough and the location problem looks solved. Networks are essential to distributed AI, and interconnection to cloud providers, data sources and partners is the reason a metro site is useful at all. But fabric alone does not run a workload.
What decides whether a site can truly host an AI workload is whether it can deliver the power and cooling that workload demands, today and as it grows. High-density systems push power delivery and heat rejection to the limits of what most existing facilities were designed for. A building with available floor space and good connectivity is still a poor fit if it cannot support the density or cannot keep the systems within operating tolerances under sustained load.
It turns out the real unit of AI infrastructure isn’t compute or the network, but a connected set of resources designed as one system. Facility location determines access to compute, power, cooling, connectivity, data, cloud services and the users being served all at once. Security, resilience and data controls have to extend across every environment the workload touches and across the connections between them, not stop at the edge of a single site. I keep coming back to the same line: AI can’t be solved with fabric alone. It needs power.
Measure the cost of a useful response
The industry’s favorite brag is megawatts deployed. Megawatts measure what a building consumes, not what businesses ROI. Most AI cost discussions make the same mistake from the other end, stopping at the model: price per token, price per GPU hour, the cost of a fine-tuning run. Those matter, but they are the cost of producing an answer, not the cost of delivering a useful one.
The full cost of serving AI includes the power to run the accelerators, the cooling to keep them running, the movement of data to and from the model, the connectivity that carries the request and the response, and the operational capacity to keep all of it reliable. Which of those dominates depends on the use case. Response time, data location, throughput and cost carry different weight for a research tool than they do for an assistant answering customers in real time.
While latency is tolerable, it’s not good. Egress and transport charges accumulate every month, and the operations team ends up managing data copies in two places to make it work. Now place the same workload in a metro facility next to the enterprise data and the cloud on-ramp. The compute is smaller and less glamorous, but the response is faster, the data stays where it belongs, and the monthly bill reflects the work being done rather than the distance it travels. Smaller or more specialized resources, or simply a different location, are often the better answer than just the largest simple available. The objective is to align infrastructure investment with the work’s value.
Build for movement
The only safe bet in AI infrastructure is that the requirements will keep changing. Models will get larger and smaller at the same time. Accelerators will turn over on a cadence the industry has never had to absorb before. Organizations will learn, through real production experience, which workloads need to sit in a central campus and which belong closer to data and users. That argues for infrastructure that can absorb change without a rebuild. Flexibility means the ability to support different generations and types of compute in the same footprint, to add power and cooling capacity in step with growth, and to connect environments as needs evolve rather than re-architecting every time a workload moves.
Think about the footprint as a whole: large campuses where scale and power are available for the heaviest work, metro facilities where workloads can sit next to enterprise data and cloud services, and the connectivity that lets a workload move between them as its needs change. Very few organizations can build all of that themselves — and they shouldn’t have to. The data center’s job is to provide the conditions and the connections that let AI workloads reach their data, their cloud services and their users, wherever those happen to be.
AI at scale will be successfully delivered by the organizations that design compute, power, cooling, connectivity and location as one system, and choose partners whose footprint already brings those pieces together.
##
ABOUT THE AUTHOR
As Chief Technology Officer at Digital Realty, Chris Sharp shapes strategy at the intersection of infrastructure and emerging technologies, from AI to quantum computing. Over the past decade, he has led the design and operation of more than 3 gigawatts of data center capacity, a scale few in the industry can match. Earlier in his career, he pioneered the relevance of colocation to cloud by building one of the industry’s first private on-ramps between a major financial institution and a leading hyperscaler, a milestone that mirrors today’s AI adoption curve.
At Digital Realty, Sharp focuses on enabling customers to unlock competitive advantage through AI-ready infrastructure capable of handling unprecedented scale and complexity. He works closely with the world’s largest platforms and enterprises to support next-generation AI workloads, advanced cloud deployments, and data-driven applications.
