Everywhere I turn, I’m either hearing about the wonderful future promised by the introduction of AI, or how it’s going to cause the end of humanity (didn’t they make LOTS movies about this?).
Will AI be used for good or evil?
I have no opinions on whether AI will be a benefit or a problem, but I do see some infrastructure issues due to AI.
And so I will state the obvious, that industry insiders are already well aware of.
Are you ready?
The massive growth of AI and data center construction projects is an economic bubble, just like the real estate bubble prior to the 2008 melt-down, the dot-com bubble, etc. And this bubble is about to pop.
Put another way, the hey-days of data centers being built with NO CONSIDERATION to their local communities, is over. It’s a zombie economy; they’re dead, they just don’t know it, yet.
Oh, I know, there’s a LOT of people on the bandwagon, pushing the latest trends for AI, how we’re gonna’ be BIGGER, we’re gonna’ be BADDER, we’re AWESOME, EVERYBODY wants AI, etc., etc.
A LOT of sleazy sales people are entirely busy “selling the sizzle, not the steak” of AI.
Industry Survey Warns of Future Trends
There are some advantages to surveys when the respondents remain safely anonymous. Anonymous surveys allow wider dissemination of sensitive information compared to non-anonymous surveys. This leads to greater accuracy. Anonymity also increases response rates and more honest feedback.
With this in mind, let us examine the 2023 Data Center Dynamics Construction Trends Survey Report.
Quick Takeaways:
72.9% said they had been able to keep up with demand, 27.1% said they were not,
Only 9.3% said they were not experiencing significant construction delays of <3 months,
19.1% said they were experiencing problems with “power connection challenges.”
The three most important factors driving site selection today are Availability of secure grid power (16.4%), Cost of power (13.3%) and Access to renewable energy (11.5%)
Other Conclusions
The final slide of the survey is easily the most important, so we’ll examine it, point by point.
This should be no surprise: This Reliability Analysis Update from PJM, dated 12/5/23, makes the power issue perfectly clear:
PJM has had unprecedented data center load growth (~7,500 MW) currently forecast by 2027- 28 in Dominion (Northern Virginia) and APS (Doubs)
11,100 MW deactivation announced
~ 5,300 MW of retirements occurred after the 2022 RTEP case was created
Now, my math may be a bit fuzzy here, but I think that adds up to a gap of 18.6 Gigawatts between new demands for data center load growth and power production centers being retired.
18.6 gigawatts will power roughly 3 million homes or New York City three times over! The ramifications are MASSIVE.
Going back to the survey, please keep in mind that while the respondents recognized that the electric grid is already suffering from declining reliability due to aging infrastructure, they were utterly unaware of the increasing gap between power capacity and demand.
Next, is this:
This is largely true because current colocation companies, generally speaking, are NOT properly maintaining their facilities to maximize availability, as I have documented in my latest essays “Save Millions by Mastering Data Center Resilience” and “The Urgent Call to Action for IT Executives: Ensuring True Resilience in Your IT Portfolio.”
The failure of colocation providers (and in company-owned data centers managed by real-estate management companies) to properly maintain their critical facilities is resulting in a “redundancy arms race,” attempting to increase resilience of their IT portfolios by having more and more failover sites, to make up for the inadequate operational stability of individual sites.
This redundancy arms race, as well as increased capacity demand is largely driving new data center construction projects. When IT clients finally recognize they’re not getting the promised reliability from colo providers that they’re paying for, the perceived immunity to the economic vagaries we suffer from will largely evaporate.
Next is the key point:
This is where the rubber really meets the road. Insiders are stating privately, what they’ll dare not say publicly; they know this can't go on.
We’ll revisit this in a moment, but the final bullet of the presentation was the following:
Power overprovisioning may be considered an “industry curse,” but it is a necessary evil in the IT world, a future limit placed on maximum electrical load in a data center, within which IT departments can operate relatively freely (provided adequate cooling can be provisioned for higher rack power densities). In other words, it’s like building a plane to carry a maximum payload; just because you don’t start with the maximum payload doesn’t mean you don’t have to include enough engines, lifting surfaces and structural integrity to carry it. You do, even though you may load the plane to only half it’s cargo capacity for 99% of its life. That’s part of the design process; asking the question reflects an inherent lack of understanding within the data center construction industry as to how IT clients operate in the real world; they don’t understand the end-users.
Given all of the above takeaways from industry insiders, plus the 18.6 Gigawatt power provisioning gap at the grid level, it’s not a surprise to anybody who really thinks about it, where we are; a completely untenable position.
But we’re not done addressing the power issue, not by far.
How AI Makes An Already Bad Situation FAR Worse
The arrival of AI and AI data-centers is exacerbating the situation.
While conventional data centers are already pulling an enormous amount of power, AI computing doesn’t use CPUs, but GPU-based systems instead, as the GPUs are tailored to better handle complex mathematical functions. But there’s a catch: they draw between 5 and 10 times more power than similarly equipped CPU systems.
This introduces a whole new level of problems, with computer racks now holding up to 100 kW of IT equipment, with corresponding heat loads. At these power levels per rack, conventional power whips and air-flow cooling are woefully inadequate. Power densities of this level require liquid cooling, and water to be the final coolant to dump heat to ambient air.
[Sadly, data center owners rarely recognize that fresh underground water is a non-renewable resource.]
This in turn is leading to some strange behavior in the conventional colocation provider market, as a response to the lack of dedicated AI data centers...
As one insider told me, a data center with 300,000 square feet of white-space (for example) with a planned power capacity of 100 watts per square foot of white space, would have a total capacity of 30 megawatts of protected power capacity. And a client putting in an AI application will take up maybe 30,000 square feet of the building, yet consume all of their power and water capability, leaving the rest of the building essentially vacant, and unused.
This is why the latest anecdotal information I’m hearing is the lack of available space for IT clients to expand within existing colocation providers.
And Now Communities Are Pushing Back
The trends will get worse, with communities now actively pushing back against data center owners building in their areas.
A quick Google search will demonstrate the trend of community resistance is growing, with MSM outlets blithely stating this to be the result of NIMBY, a “Not In My Back Yard” mentality. Such shallow dismissal of local communities hides the much bigger problem of diminishing power capacity while data center owners absolutely demand more power. And when those same communities realize that the AI hardware based on GPUs will also require the communities to sacrifice the water for their own families…?
I think the following meme sums it up nicely:
Let me ask you a question: WHY are the big IT companies going out to buy land in the middle of "nowhere?" Why is Facebook buying Farms in Nebraska, Microsoft buying farms in Ohio, or Amazon buying 430 acres, 40 miles outside of Atlanta?
It's NOT because there's a lot of fiber pipeline in a corn field, and it's NOT because of easy access to international airports and shipping hubs... It's because water and power are available, the local leadership can be easily bought with the promised increases in tax revenues. Basically the local leadership is too naive to see far enough down the road to the reduced ground-water and grid reliability for their real constituents- their local populations.
I predict that soon, the massive amounts of money dumped into all these projects, with no accountability and no responsibility, are going to be turned off, whether voluntarily or through legislation forcing moratoriums, as has happened in London and Germany.
Conclusion
I've said for a long time that data centers are not "environmentally friendly," they're not "green," or anything of the sort. With current technology, they're energy HOGS, plain and simple.
So whenever I've heard someone talking up "sustainability" and "renewable energy," this is what instantly comes to mind:
GPU-based AI systems, which are even worse with regard to efficiency and "sustainability," take that energy pig and increase it by an order of magnitude. It's10x bigger; it's an elephant.
And it's going to be living in your backyard.
The bottom line is quite simple: GPU-based AI systems are not sustainable, they're not efficient, and adding this to the already dicey situation regarding power and water for enterprise data centers and colocation facilities means the situation cannot continue much longer.
While I have no opinion regarding the actual utility of AI, it is my professional conclusion that GPU-based AI systems are NOT the answer.
Going further, GPU-based AI is a dead-end, because it's something our infrastructure simply can't afford to support.
Industry leaders must have the courage to face this fact, instead of "selling the sizzle."
We need BETTER solutions, than GPU-based AI.
