# AI Infrastructure Bottlenecks: Energy, Memory, and Optical Shifts

**Podcast:** a16z Podcast
**Published:** 2026-10-09

## Transcript

In an AI digital age, energy capacity equals economic capacity.
Why build a new data center and buy a million GPUs if I can't power that?
You're going to see more and more defaults happening on many of those data center projects because the energy won't be there.
Whenever you have the technology to make something easy, that means the bottleneck moves somewhere else.
Nothing's a chip anymore.
It's a rack.
It took me three months to design it, but it's nine months until I can actually...
start to use it.
Exactly how many major new memories have you had over the last 30 years?
Zero.
Memory innovation for the first time in 30 years is nigh upon us.
I declared the death of copper about 25 years ago.
Eventually I'll be right for all of us hardware guys.
It's like renaissance in front of AI inference accelerator chips.
And I'm sure I don't even know them all.
Why are you guys funding so many of those anyways?
Historically, there have not been a hundred competing processor vendors in any industry ever.
Is this a temporary thing?
It'll convert back to a few.
I see it as...
AI is making it faster to design chips, but actually building them, powering them, and connecting them may be getting harder.
In this episode, A16Z's Ragu Ragu Ram and Guido Abedzeller sit down with Pat Gelsinger, general partner at Playground Global and former CEO of Intel and VMware, to discuss the next bottlenecks in AI infrastructure.
Pat takes us back to his early days designing Intel's processors.
when engineers were inventing the tools they needed to build the chips themselves.
He explains why AI could bring a similar shift today, even as manufacturing timelines, memory bandwidth, and power become harder constraints.
They also debate whether the explosion of specialized AI chips will last, why Pat believes memory innovation is finally approaching a breakthrough, and how optical networking could change the architecture of AI clusters.
In drawing on their shared history at VMware, they close with a question.
What would it look like to rebuild virtualization for a world where AI agents, rather than humans, are the primary users?
I'm here with our esteemed guest and dear friend and former boss, Mr.
Pat Gelsinger.
Welcome, Pat.
Pat is currently the general partner at Playground Global, but as you're very, very well known in the industry for leading Intel.
for being the CTO of Intel, of course, leading VMware and many other things.
So welcome.
Hey, thank you, Raygu.
Great to be with you and Guido.
And to me, this feels just a little bit like old home, right?
We superimposed a year of change, right?
But you both went the same.
I feel energetic.
So let's dive in.
There has never been a day when you stopped being energetic.
So there's no news there.
But yeah, no, let's start actually from your Intel career.
Recently, Andreessen Horowitz created this.
Horowitz and Reesham Academy, which was to find talented people between 16 and 22, and then put them into a modern educational setting so they'd be well prepared for either doing things of their own or joining the companies of today.
You had a similar experience, like you went to a regular trade school and then jumped into Intel when you were, what, 18 or 19, I should say?
Yeah, it was really a sort of magical period.
I'm 16 years old, I accidentally win a scholarship, I go to tech school, I skip my last year and a half of high school, and Intel comes recruiting when I'm 18 years old.
And I've never been on an airplane.
had already fallen in love with computers at tech school.
And the interviewer, right, Ron Smith was his name.
He writes on his page.
And I was number 12 that he interviewed that day.
And if you've interviewed 12 people in a row, you can't tell male from female by the end of it, right?
I'm number 12.
He says, smart, aggressive, arrogant.
He'll fit right in.
So I got invited to come to Intel and it just became really glorious.
Starting as a technician, moved into the design team at the end of the 286, engineer number four on the 386, architect and design manager for the 486.
All of that while doing my bachelor's, my master's and PhD work.
So it was like the best career that you could possibly have.
You're learning by day and you're putting it to practice at night.
That's right.
That's right.
I've been creating a completely new era of chips.
I bet most of what you learned.
And what you built, there was a big gap because you're breaking new ground.
Yeah, in a lot of ways.
And it really was a pretty incredible time in the industry.
And I remember one of the professor classes, he postulated a new Cary Lookahead Adder design, right?
In one of my classes at Stanford.
So I said, well, I'm working on the 386 Cary Lookahead Adder.
So let me go try it.
So I come back two days later to the professor and I said, it doesn't work.
Right.
And I started an argument with my professor at Stanford.
You know, it was like the next two class periods were just me and the professor arguing over why his design didn't work.
He had a book that was being published with that design in it.
So it was very annoying.
And we started built-in self-test.
Right.
And I remember Ed McCluskey at Stanford was my professor.
And he was sort of the father of that.
And we had wonderful arguments around why most of his ideas were bad.
They weren't practical for real.
chips, but I was putting built-in self-test to work on the three and six.
John Hennessey was my thesis advisor, and obviously he did okay.
I helped his career.
He became president, and he's done okay since then.
Later on, we'll talk about the CIS courses.
I had the same experience when I was teaching at Stanford.
You come with something that you think is the truth, and then the student comes.
that works in industries.
Well, actually, not quite.
So the next lecture you come back with, okay, I looked this up, you're right.
This is outdated.
Yeah, and I'll tell you, you know, that's part of the, you know, that interaction, and I think that's part of what makes Silicon Valley so unique.
Yeah.
Right, you know, is you have so many companies and entrepreneurship.
It's like in the water here.
Starting things up and new ideas and challenging the professors.
And you can't be a professor at a place like Stanford and not expect to get challenged and enjoy it.
There's good foot traffic in and out of that ivory tower.
Yeah.
Yeah, now, 486 was your big accomplishment, if you will.
Obviously, lots of people worked on it.
And you guys broke some new ground in ship design there, right?
and increase what functions the chip was supposed to do and so on and so forth.
Yeah, it was a pretty magic period as well because it was before what you would think of as the EDA industry.
Oh, wow, yes, that's right.
And at the end of the, a little bit of the 386, but the 486 was really the first chip.
to implement what you would think of as modern EDA techniques.
We did a high-level design, right, a description, an RTL description, but there was no Verilog.
So we invented HDL, the Intel hardware description language.
So I wrote my own language.
Well, you had to build a compiler for that language, right?
So we created a compiler for the language.
There was no automatic placing route.
So we had to invent that.
And we worked with Alberto Sanjivangi, Vincentelli, Berkeley, and some of his students for the first place where the first routing, the first automated timing management.
So they created EDATA to some degree.
Yeah, to a great degree.
And that was really one of the hallmarks that not only was the 486 this compatible pipeline microprocessor, but we ushered in many of the foundations of what became the modern EDA industry and was really a pretty magical period of the industry.
Yeah, yeah.
If we look at today, will people look back as Blackwell?
and saying this was of the last generation that was done before AI took over the design process for Microsoft.
Are we in a similar transition right now?
I think there are certainly aspects of that.
And I think when you look at things like Jalapeno today, right, you sort of say that's sort of a first principles use of AI in the chip design process, right, where you throw out a lot of those things.
Now, there are pieces of design that are still really hard.
Right.
In that sense.
And a lot of the analog, right, aspects, certes is probably the best example of that, you know, are still not AI-able.
Right.
You just need lots of silicon data.
Right.
To get those.
So you either get so conservative in your analog design that you're able to, I'll say, AI it or, right, you got to do those the hard way.
But now many of the logic functions can really be done with AI tools and techniques.
pretty incredible ways, right?
Your transistor budgets are big enough.
The design tools are getting smart enough.
A few experts guiding the tools and how to apply it.
And I really think it will be somewhat like the 46 that way, right?
We just sort of look back and say, yep, that was the beginning of a new era of chip design.
Whenever you have the technology to make something easy, that means the bottleneck moves somewhere else.
Can we guess at this point where the bottleneck will be in the future?
There's some things which it seems like AI is incredibly good at, right?
Like creating the software layers, you know, writing kernels, a lot of the, I think a lot of the sort of being able to specify the objectives at a higher level, which then gets sort of translated into a very long, right?
And so what is the new frontier?
What is the thing that will be difficult going forward?
Well, when you look at today, so let's say Guido and Pat, we're off to do a great new chip together.
We're going to...
Right, right.
And it's the one that's going to leap ahead because we have understanding of the AI...
We have understanding of the AI workloads.
We have understanding then of how to compose that and then the right multiply, accumulate, register structure.
Perfect visibility of the model workloads of tomorrow as well.
Yeah, and all of that kind of stuff.
And let's say we turn our AI tools loose.
We are great.
Within three months, We have an awesome design, right?
For that, we've been a little bit conservative on all the analog components of it.
Now, we still have a couple of really major bottlenecks today.
One is, right, we still have nine months of silicon processing time.
Yeah.
Right, something like that.
So I can design the thing in three months, but I can't actually get it into real silicon at scale for nine months.
Okay, that sucks.
Right, so we have to really see that bottleneck and prove if we're going to have this innovation because we know.
Are there validation and verification stages or even beyond that?
Well, you know, first.
I can't get anything out of fab, you know, in less than three months.
And that's if I have, you know, like supercharged design flows, right?
So I said, you know, how do I compress that, right?
You know, for it.
And then I got to get into advanced packages, 3D packages are getting complex.
So that's like another three months, right?
You know, and then I have to put it into a rack scale solution because nothing's a chip anymore.
It's a rack.
Yeah, so we're nine months.
You know, it took me three months to design it, but it's nine months until I can actually start to use it.
So how do we start to compress those aspects of design?
And I think we need new forms of lithography to enable that.
We need design flows that don't require $50 million of mass cost until I can get things into prototyping.
So to me, how do you compress that to a month or two?
Because I can now do the design in three months.
How can I have that done in a month or two?
Because if it takes me a year and a half until I actually get scale and software on it, Okay, my understanding of the AI workloads is no longer applicable to the chip that I designed, right?
You know, so, you know, all of those...
We've seen that play out live right now, I think, with the AI chips, yeah.
Right, you know, it's almost, you know, and we can look at it historically.
It was like, you know, Graphcore wasn't a bad design, but the world moved on, right?
You know, in that.
So all of those aspects somewhat obviate the fact that AI has made the design piece easy.
all of those manufacturing scale issues to me become a huge bottleneck, right?
Another bottleneck, of course, is memory.
Yep.
Right, you know, memory bandwidth.
We noticed, yeah.
Right, and, you know, for that, as I've described, HBM is a hideous memory.
It's just the best one that we've got.
It's a terrible memory.
You know, I get terrible bit density.
I'm limited to shoreline bandwidth, power issues, thermal issues, DRAM doesn't like to be hot.
Concentrating everything in one spot.
Yeah, it's terrible to give an AI as a memory.
compute workload.
Yes.
Right.
So, right, you know, for that, so I, you know, I need to, you know, compress that development and embodiment into real scale.
You know, I need to fix memory, right, associated with this.
So, to me, that's another critical bottleneck for us to attack.
And power, right, you know, power in, you know, thermal, and most of the power simulation aspects today are bad.
Right, you know, you don't have good 3D modeling tools for power, power dissipation, hotspots, et cetera.
You know, my voltage in, right, you know, I'm essentially guard banding almost my full power envelope.
Correct.
Right, so essentially I'm losing, you know, 40% of my power in guard banding, so I got to get much better at power modeling, you know, and then my power dissipation techniques are bad, right?
So, you know.
Well, every model has to change too.
Yeah, so all of these things, you know, to me, those are the next big bottlenecks, right?
You know, how do I improve each one of those?
And now, right, you know, if I've done that, okay, now we're on to an exciting era for what's next.
So, you know, thank you for those great AI tools, but, you know, I got big challenges even when those are fully deployed.
And so we've seen sort of a rebirth in the number of chips and chip architectures and so on and so forth.
Yeah, yeah.
Thanks to these workloads.
It's like there's 100 AI inference accelerator chips, and I'm sure I don't even know them all.
Yeah, exactly.
Why are you guys funding so many of those anyways?
You funded your fair share, too.
Yeah, so they're all, to their credit, making different design assumptions, either in the components that you talked about or in the part of the pareto curve that they're targeting.
So do you think there's room for all of these?
Or are we in a world where, partly because of what Guido talked about, AI tooling, we now have the ability to build more different kind of chips?
Yeah.
And the time is enormous, whichever way you want to look.
To us, it's even more provocatively.
Historically, there have not been 100 competing processor vendors in any industry ever, right?
So is this a more heterogeneous future?
Will we see, is this a temporary thing?
It'll convert back to a few?
Yeah, and I see it as a more temporary thing and it will converge is what I expect.
And there's probably three different reasons that I see that to be the case.
One of them is this emerging heterogeneity.
that you see in the compute environment where, okay, now I have specialized pre-fill versus decode.
Well, you know, and now people are saying, well, we really need a specialized tier for mid-fill, not just pre-fill, right?
So now I have early pre-fill and now I have mid-fill, which is, you know, different.
Well, maybe speculate and verify if we listen to OpenAI here, right?
Yeah, you know, and all of a sudden my compute fleet becomes more and more granular across the workload.
And I think any time that you've seen that in history is not sustainable.
And then all of a sudden, right, when, you know, as you look at that, and now people are saying, well, as I go to reasoning models, I want something that looks more like a CPU again.
So, you know, all of a sudden I didn't want that level, you know, that didn't become the predominant portion of my compute capacity.
I needed more of it over here, right, and model and model workflows and, you know, so on.
You know, so I really don't like...
I'll say at-scale specialization, right?
Because I think the workloads are going to continue to moderate, you know, and migrate so significantly.
First, you know, second is I think today's models are about to go through some evolutionary breakthroughs as well, right?
You know, I think the LLM is sort of reaching the limits.
And now as people look to how do I really model 3D?
Right, you know, where, you know, a flattish LLM doesn't work well when you go to molecules and chemicals and, you know, imaging.
So I think we'll see limits of that that will cause different shifts in the algorithmic domain.
I also think that some of the most interesting workloads become where you start, you know, I'll say bringing HPC-like things back into AI-like things, where all of a sudden, you know, 64-bit precision counts again.
What would be an example of that?
Well, imagine that my AI model is now inducing the five most interesting domains for a chemical analysis.
Okay, those chemical algorithms are going to run with high precision, which is going to be the dominant piece of the workload, getting to the five algorithms I need to run or running the five algorithms.
on different chemical or biological systems.
So I do think that as we optimize our AI systems, they will lead us back to some of the things that have been more traditional high-performance computing.
So I see that aspect of workload as well.
So all of that said is this extreme specialization to me in the compute architect.
I sort of don't like it because I don't think it's going to be what the workloads will look like two, three, four years from now.
So that's one reason I don't see all of these AI chips as they get more and more specialized for portions of the compute workload to necessarily be right.
You know, second is, right, there's like 100 of them now, right?
You know, there's multiple optical ones, there's probabilistic ones and so on.
And you know they're not all going to win.
And they're not all going to win because at the end of the day, you have to get scale.
Yeah.
These things and scale requires, you know, you have to win, you have to get capital, you have to get workloads onto it.
So I think it sort of defies logic that you're going to have a hundred of these things.
So I see then.
you know, coming back as a result of, you know, simple, you know, capital market share, et cetera, you know, which are the winning teams, you know, winning designs, winning architecture.
So, you know, I see the workload driving that, you know, I see the natural, you know, industry consolidations, you know, that will occur.
And then I also expect that the winners will pick some winners, right?
You know, where, you know, an open AI, an NVIDIA, an Anthropic will say, I like this one.
Because it isn't just hardware, it's how do the hardware and software code evolve.
Right?
You know, for it, and I do expect that there's going to be, hmm, you know, here's this 10 I could pick for this.
I like that one.
And it does require investment in the software and the workload evolution to take advantage of those platforms.
So for those three reasons, yeah, we're going to see, you know, a narrowing of the field.
And that's...
And the deployment complexity as well.
Sorry, go ahead.
Makes total sense.
And I think I agree in most part, let me still try to stir a little bit of controversy here just to...
Are you, Guido, stirring controversy?
No.
You guys have changed it, right?
So obviously...
100 companies won't access or consolidate, right?
And I completely agree with your argument that at the end of the day, the large consumers will pick winners to some degree, right?
Today, I think to be a first-tier chip startup to some degree, whether you're that or not, is defined by how much traction you have with the really big guys.
That said, and there's also an argument to say, like, look, in any kind of transition period, you probably have a bit more of an explosion of different approaches.
Now, all that said, I think there's an argument you can make that basically says, look, historically, we standardized on, you know, one or two architectures because, you know, different instruction sets were hard.
You had to build multiple compilers, you know, like the, it was basically the software complexity dictated that you can only have a certain small number of hardware platforms.
That part seems to be changing with AI a little bit, right?
Today, you know, writing an optimized kernel.
I mean, I think jalapeno was a great example, but they basically said, look, humans can't really program this thing anymore, but an agent swarm overnight can do it.
That's all we need, right?
So basically you can say, look, if you can build a chip that has a certain capability, it's now much, much easier to build the software layers on top that fully leverages that.
Could a result of that be that we have a little bit more heterogeneity?
Just because, you know, it's easy to program that heterogeneity goes through.
And I do think it changes some of the, you know, where the bottleneck lands, right, changes in that regard.
You know, the first bottleneck is the one I'd point to that we've already talked about.
Still takes a year and a half to build these things.
Yeah.
Right.
You know, and then the second bottleneck is not only does it take a year and a half to get these things into racks and scales and so on like that.
Now, to make them useful, okay, you know.
Could I borrow 50 billion?
Right.
I mean, you're not talking about small capital expenditures.
You're talking about very large capital expenditures and data centers, you know, power, you know, commitments and so on like that.
I got to build this thing at scale, right, associated with it.
So that's going to defy a lot of heterogeneity as a result.
Now, you know, clearly big players.
And we saw this with what, you know, NVIDIA did with Grok.
Okay, you know, we're going to sort of own some of that heterogeneity.
Right.
And, you know, two years from now, you're not going to tell what Grok is inside of NVIDIA.
Right.
It's just going to be like another special.
Right.
You know, someone like that.
So you'll see the architectures, you know, sort of hide the heterogeneity right inside of it, which to me is just fine.
Right.
You know, I mean, that's what layering, that's what abstraction has always done associated with it.
Yeah.
But I think those forces are powerful ones to, I'll say, expose a lot of the heterogeneity.
No.
Right, I think it will get abstracted underneath some of these.
But at the end of the day, it's still hard to build these things on scale.
Earlier you talked about HPM being hideous memory.
By far the best one we have today.
Exactly.
The thing about democracy.
We can't recall when there was the last memory innovation.
It might have been obtained.
I think you probably had something to do with it.
Yeah, that one died.
That one died.
We killed it.
Yeah, we killed it.
Do you think memory innovation is around the corner or what needs to happen for, obviously we understand the scale problem because it's even more larger there.
But if you keep the scale aside, even from a technology point of view.
Yeah, and let's, you know, look at that history.
I've probably personally been associated with at least five different new memory architectures, Optane just being one of them, right, you know, that did not see the light of day.
right, as well.
I'm probably familiar with close to 100 that have happened in the industry over the last 30 years.
And exactly how many major new memories have we had over the last 30 years?
Zero.
Right, which, you know, okay, DRAM, SRAM, flash.
Okay.
What else is there?
DRAM, SRAM, flash.
You know, so I do think...
Flash stacking comes for a little bit.
Barely.
Yeah.
Right, you know, but anyway, right.
So, you know, in that, you know, it has been, right, you know, a disappointing that we haven't created, you know, the next physics that enable, you know, low-cost, high-performance, resilient, manufacturable memory, right, you know, for it.
So I do think, and now, and over the last 30, 40 years, you know, memory was a terrible industry, right, because you would make money one year and then you'd lose money for the next four.
Right.
So let's go do R&D in an industry that makes money one out of five years.
Right.
You know, it's just always, you know, it's just been so terribly difficult because it's been such, you know, driven by such violent commoditization cycles.
That has changed a little bit though recently.
Over the last five years?
Happy days, man.
So all three big memory of Amazon.
in the top 20 most valuable companies on the planet.
That's a crazy change.
It's just so, right, you know, who would have thought, right, associated with it.
And anybody who looks at the AI workloads, and it's always, you know, it's much harder to, you know, it's easy to predict the past, but it's a little bit harder to predict the future, right?
And, you know, in that, you look at it and say, AI is a memory workload.
Right, you know, inside of it.
So I think there is the combination of capital.
and technology needs that will justify memory innovation.
Now, there's two domains that I'm excited about, you know, for it.
You know, one is, you know, this idea, you know, I do think that the shoreline bandwidth limitations of HBM is a fundamental issue.
Yeah.
Right.
So with that, the idea, you know, and obviously, you know, we have a company, Dmatrix, that's doing this, but I think others will be, you know, whether it's, you know, Cerebris, you know, and their techniques.
The future is stacked, yeah.
Somehow you got to bring memory and...
In more ways than one.
You know, you got to bring memory and compute together.
Yeah.
in fundamental new structures.
I do think that given the workload now has such capital being deployed against it.
in the AI space, that I do think will actually be able to break through some of those physics challenges as well.
So I do believe, and, you know, I've just funded a new memory company as well.
That's still a stealth, but, you know, I'm quite excited about some of the innovations that will occur in new memories.
And you're going to see new materials, you know, for memory, you know, a lot of people looking at things like ferroelectrics, you know, finding non-capacitive.
you know, high-density memories, finding memories that can, you know, be high-performance, high-density, and stackable, right?
But, you know, people are looking at low-cost flash, you know, for that.
I'm not particularly enthusiastic about that idea myself.
I think, you know, speed and performance of the underlying cell are somewhat problematic with a flash or an MRAM structure.
You know, but I do think there's memory innovations are coming for the first time in 30 years.
And I do think that's going to be an exciting space.
And, you know, we'll have a couple of companies there, I'm sure.
Yeah, hopefully we'll do a few of them together.
What do you think?
Should we do?
Okay, you know, but we'll find some things to do here.
But I think memory innovation for the first time in 30 years is nigh upon us.
Yeah, yeah.
It's amazing what a couple of $10 billion of market cap got to get 100 billion.
A trillion?
Okay, sorry.
You know, I think the memory industry has now increased by market cap by $2.5 trillion over the last four years.
I think that's about right.
You could have just invested there and not done any of your financial stuff.
Yeah, exactly.
Use your money.
How long?
What's your view?
Sorry, go ahead, Guilla.
To expand a little bit on the stacking part, if you have to predict, how quickly will our chips get taller?
Is this going to be like, you know, we're going to go to two and then sit there for a while?
Or will we have, you know, the 16-layer sandwich, you know, happily mixing logic, memory, and SRAM?
HPM pulled out this.
HPM is going down at Moffat.
But how do you see this evolve?
Yeah, you know, I think there's, you know, the problem with stacking, right, is now essentially the value of the stack.
right, becomes greater as you go to higher dimensionality of the stack, which means the yield of any individual layer in the stacking process has to keep coming.
You know, essentially it can't go up linearly, right?
You know, it has to go up exponentially.
And improving the manufacturing and defect characteristics in an exponential way, you know, when you get to eight or 16 high stacks gets to be so, so hard when you run the math on that.
associated with it.
You know, your manufacturing flows have to be so close to perfect.
Although you could see designs that take into account a certain model class, like we've seen it for a long time for two-dimensional.
Yeah, of course.
But remember, that was the nature of DRAM.
You always had bad block replacement and, you know, spare lines and so on like that.
So not in any way, right, you know, minimizing that.
But assuming you're doing that, you still need the yields to be so good.
You know, if I have a cracked die, I don't care how much resilience you've built into it.
You can't.
you know, it's a bad die, right?
So, you know, those kind, you know, so as a result of that, I'm not, you know, a crazy 16, 32 stack, you know, kind of guy.
I think there's going to be, you know, nice three, four, five stacks, you know, that just end up sort of being the sweet spot.
The mid-rises?
Yeah.
Right.
You know, for and I expect for, you know, part of its manufacturing, part of its complexity associated with it, you know, where's those yield sweet spots associated.
So that's sort of where I think most things land.
But even there, right, you know, so let's say I now have, you know, I now have a great compute tile.
Right.
You know, how big does that tile need to be?
I mean, everything's become reticle limited now.
Right.
You know.
Can I go to more chiplets that better yield characteristics, better thermal?
And then you're going to say, I want a memory stack on that.
Is it a two stack memory, a four stack memory?
I don't see it becoming an eight stack memory.
So to me, two or four is sort of right.
But then power delivery needs to get integrated into that.
So that becomes sort of another element.
And I want that in the Z dimension.
associated with it because I want my power rails coming into this as well.
And then I need fairly intelligent RDL layers to distribute signals across that platform.
So to some degree, my canonical thing is already about eight layers thick when you add up all of those pieces associated with it.
And I think that's about all you can do, right, at least as we understand manufacturing today.
And then, of course, we expect optics to come in, so you're going to have more 3-5 material structures that are going to be closer to this complex because I want my I.O.
getting essentially deeply integrated into that construct as well.
So my silicon-centric pieces need to get complemented by some of these other 3-5 materials and how do I get optical connectivity built into it.
this?
How do I get power rails built into that?
That is a, I'll say, that is an engineering feat and a manufacturing nightmare to bring all those pieces.
But that's where the physics is taking us over time.
So because of that, throwing a 16 high stack, it's like, that's crazy.
A two or four high memory stack, I think that's probably where the sweet spot will be.
And you didn't talk about the thermal dissipation.
And new materials needed for that.
Yeah, yeah.
And obviously there's going to be a lot of, you know, first thing, right, you just can't have some of the, you know, the physics hotspots that you have today.
So you have to be more intelligent about how you use the heat and spread the heat.
But, you know, then you're going to be saying, okay, how do I both get power in and dissipate power out?
of those environments.
And by the way, if I can get much more efficient power in at the voltages I want with much better power responsiveness and my voltage regulation, then I don't create as much thermal chimneys.
But I do have to think, okay, how do I cool these things?
And, you know, what role do new materials, diamonds, and other things have been postulated as ways to, you know, do a better job of the thermal cooling?
How do I then have better, you know, liquid cooling techniques and others, you know, sitting right on it, you know, with good thermal characteristics?
So, yeah, you know, we engineers are becoming plumbers, right?
Exactly.
Yeah, you had to worry about MEC, the chip scale.
Let me ask you about an operant phenomenon, which is some people have said, hey, let's get the memory off of the package and just go all optical between the compute and the memory.
And that'll give you more degrees of freedom and more ways of building things.
What's your view on that as opposed to stacking?
That's the other extreme.
Yeah, you know, I think the more OEO, right, interfaces that you have, you know, it's all.
optical to electrical to optical to, you know, the more of those that you demand in your core compute complex, you know, to me, it's just hard, right, in that sense.
And you're always going to have losses associated with it.
You know, people are, you know, working hard to come up with more clever techniques to decrease, you know, both the cost and the power.
How much of the power am I wasting as I go through my DACs and conversions?
So I'm not a fan of those because I think it gives you access to large pools of memory, but I burn a lot of power getting there.
And remember, essentially, the femtojoules per compute is now actually...
pretty amazingly good, right?
The femtojoules per bit of communication is comparatively a thousand times worse, right?
And if I'm spending that much to go get to large pools of...
So I don't like those approaches.
You know, myself, I think I somewhat defy the underlying physics of the solution.
I mean, and then there's sort of the other approach to say, like, look, can we actually integrate compute and memory directly, right?
Shift some of the compute into the memory.
Some of the PIM approaches.
Yeah, exactly.
So what...
What are your thoughts about that?
I'm not a fan of those either, right?
Because I think workloads, you know, because now I have to take my workload and I have to like push these little pieces of compute into the memory, right?
You know, so that, you know, I don't need as much bandwidth of memory coming into my real compute, right?
And that puts constraints on workloads.
And again, you know, I could be wrong on this, particularly some of the design flexibility and the intelligence that we can now bring, you know, through the design aspects.
But most...
PIM ideas, you know, have been around for 25, 30 years.
And I think 25, 30 years from now, they'll still be hanging around, right, in that sense, you know.
Give me great memory bandwidth, you know, that I can have, you know, high density, high performance, and get it close to the real compute engines.
I expect that's going to be a more generalizable solution across many and evolving workloads over time.
So we're still stuck with...
Denser and denser chips, more compute, more logic, more power.
Much better I.O.
I am a huge fan, right, of moving optics for all I.O.
function, right?
I declared the death of copper about 25 years ago.
Eventually, I'll be right.
You're still right.
One day soon, I'll be right.
But I think is that, you know, to me, all I.O.
should go to optical, right?
But that core compute memory complex, no.
So, Wendy, the first step for going on optical is to scale up, even more so than scale out.
Yeah, yeah.
What's your views on that?
Yeah, you know, and I think we're now seeing...
What would be the application of the workload driver, if you will?
Well, I think the workload driver is the one in front of us.
Right.
I mean, you know, today, you know, our scale up environments, we're putting tons of copper, right.
You know, and essentially copper is becoming a wave guide that's getting shorter and shorter.
Right.
And all of a sudden, you know, to make my copper work at, you know, five meters is more expensive than making optical work, right, at 100 meters.
Right.
You know, so, you know, I think the physics is driving us there today, you know, and, you know.
So if that's the case, Pat, why haven't we moved there already?
It's a complex supply chain that has never been tested at scale.
And I think now, I think everybody is looking at some form of NPO, CPO, etc.
in the 28-29 timeframe because I can't scale up to large radix, compute clusters for my scale-up environment without making that move.
So clearly, NVIDIA has indicated that.
I think everybody else in the industry expects that's the conversion point.
In reality, I should have never built an NBL 72.
You know, it's an engineering marvel and it's a manufacturing nightmare, right?
You know, it took the industry, you know, one extra year and a half to digest that beast, right?
And it really is amazing that, you know, we're able to get it to scale and so on.
But, you know, that really proved that had we had a more mature optical supply chain, we'd be much more advanced today.
We'd be able to build better scale-up environments today, you know, better, you know, power, better cost, better energy.
But we didn't have the maturity.
of the optical supply chains to allow that.
And those things are not trivial to get resolved.
There's a lot of work to happen, but I think 28, 29 is the year.
Fab capacity.
We're probably gaining a lot of experience how in-packaged optics work in networking, right?
And I think many of that will translate nicely to GPUs.
Yeah, I do think.
But all of a sudden, you know, it's like, okay, these things have to work, you know, at 10 million GPU scale.
Wow, right, you know, and it has to fit in packages.
And, you know, there's just a lot of things there, you know, and many forms of optical don't like bad thermals.
Well, guess what?
Bad thermals is the nature of the game for it.
You know, so we have thermal problems to deal with, you know, package integration problems.
We didn't have enough, you know, laser capacity in the industry.
So a bunch of things, but I don't see anything there fundamental.
It's just a lot of hard work at this point to pull all of those pieces together.
In terms of the networked architecture, there's also...
mean a switch, what's more circuit-switched networks, just because, you know, you can't scale these very fast optical networks indefinitely.
Well, you know, I like, you know, Nick McCune, you know, another Stanford professor here, good friend of mine, you know, as well.
You know, I mean, you know him a little bit also, Guido.
He was my thesis advisor.
Yeah.
But, you know, to me, he describes it well.
You know, we built, you know, networks to be able to handle any packet going anywhere with no knowledge of where it might go.
When you think about AI, you know, it's almost exactly the opposite.
Right?
You know, we're doing flows that are large and predictable.
So, in that sense, just the very description says, hmm, that looks more like a switch than like a packet.
So I do think this confluence of moving to optical, right, and, you know, having a workload that is far more predictable, manageable, you know, large in flow, you know, to me, you know, I do see that, yeah, we will go all the way to bringing optical into the compute complex and we will go all the way to OCS or some form of optical switching, you know, with more predictable flows.
You know, that is the right.
you know, architecture.
And then you sort of say, is that scale up or scale out?
And to some degree, if my scale...
Yeah, I was going to say, it sort of dissolves the boundary.
Yeah, right.
If my scale up is big enough, then yeah, you know, just saying, okay, what's the radix in my scale up environment?
And when do I want to hook it up to other clusters that are really big?
And to me, that ends up being a pretty beautiful answer, you know, to the future.
I'm looking at somebody who's way more expert on this than myself.
Look, I always found scale-up was a scale-up to some degree an artificial distinction.
You understand where it comes from, right?
We're kind of using different protocols.
One is front-end, one is back-end network, and so on.
So right now, practically speaking, it is a big difference.
If you take a step back and look at it from a system architecture perspective, it achieves the same goal, you know, with slightly different flavors, slightly different protocols that over time I think will converge.
I mean, especially for the training workload.
Yeah, although training and inference is actually look fairly similar.
The inference corpus is less similar.
Yeah, now I'm going to, you know, right, does your brain stop learning?
Right, you know, by the way, as another characteristic of workload evolution, I do think the separation of training and inferencing is going to become less so going forward, not more so.
Right.
That's a little bit of the argument against, you know, you know, more and more heterogeneous architectures is I think continuous learning, you know, I think there'll be algorithmic domains that will say, yes, continuous learning, you know, becomes the nature of the algorithm where, yeah, I do want my inference environment to be updating my model weights because, you know, that's giving me more and more refinement.
you know, evolution across, you know, workloads, moving into environments that are less formalized training as well, we'll have smaller data sets in those environments.
So, you know, I'm not sure that's a...
But the number of nodes you need in order to be useful is a smaller diameter for inference than training.
That's what it means.
Today, that's true, yeah.
I mean, I still think there might be a role for heterogeneity coming from a different angle, which is that different...
types of AI workloads require very different underlying good.
Like some are more compute limited, some are more memory bandwidth.
They have a small diffusion workload also, right?
The optimal card for that looks very different from, you know, top of the line, you know, $15 trillion, sorry, $15 trillion weight LLM, right?
That means you're creating massive scale.
So I think that might drive heterogeneity, if anything.
Yeah, I think there's some room for that.
But I'd also say, okay, now I'm talking about putting a million GPUs to work.
in my data center.
And you say, well, for that particular diffusion workload, I should really put something optimized this way.
So I'm now going to start putting 1,000 of those in.
And for that workload, I need 2,000 of these in.
And all of a sudden, you don't get some of the flexibility of scale that you would say.
And how much does it cost me to manage and upgrade and connect?
and your fault domain, all those types of things.
The agents will do all the upgrades.
That's no longer a problem.
Thank you.
Everything gets easy.
So good out of here.
So fleet management is done.
So we talked about what?
Compute, memory, networking, power is the next one.
Yeah, yeah.
And you touched upon it with vertical delivery inside the chip, on the chip.
I mean, there's 800-world DC, there's power delivery to the data center.
There's an entire ecosystem all the way to power generation and the sociopolitical dimensions of that and so on and so forth.
Yeah, yeah.
So where does innovation come in and where does, like, it's pure execution?
Well, the, you know, and if we start at, you know, first principles here.
Our nation has done a terrible job with its energy capacity, right?
You know, essentially we went through 10 to 15 years where essentially I was taking coal offline at the rate I was adding renewables.
And essentially the nation was flatlined in terms of overall energy capacity.
Not good.
Because in a AI digital age, energy capacity equals economic capacity, right?
So essentially my economic capacity as a nation, flat for 15 years if you buy that thesis, which I think is very, you know, provable.
Over the last five years, you know, we've seen this enormous influx into just giving more capacity.
But, you know, I think, you know, even with that, we've gone to maybe increasing our national energy capacity 4% per year.
Something like that.
Going from, you know, essentially zero to one, right?
Wow, 4x, right?
Or, hmm, 4%.
Right.
So to me, you know, that, you know, it's a bad situation.
So we just need more energy capacity for it.
So number one, and obviously we have some companies in that space.
You know, we talked about nuclear operating as one of those with Alva.
And, you know, I think nuclear, right, you know, as a baseload is, you know, a fabulous thing that we want to go build more of.
You know, unfortunately, all of our renewables have.
you know, deep dependencies in China, which is very unfortunate, you know, for us, you know, we're, you know, you know, gas turbine lead times are only eight years at this point, right?
So, you know, you know, I mean, we really have a difficult environment to scale up rapidly, but one, we just need more capacity.
And fundamentally, energy capacity somewhat dampens how you fork we can get on AI.
Right?
Because, you know, why build a new data center and buy, you know, the million GPUs if I can't power them?
Right?
And I think you're going to see more and more defaults happening on many of those data center projects because the energy won't be there.
So you think it'll be a significant headwind?
I think it will be a headwind.
And I think you're already starting to see some of the first indicators of that, the Oracle, you know, was just, I think that's the first of what you're going to see many.
Now, it's not like we're going to slow down as a result, but I think people will start saying, do I really have the energy, you know, to underwrite the capital commitments that I'm making, you know, for, you know, cement, for data center racks and GPU purchases?
So I do think it's, you know, going to be, you know, an upper bound.
But we need more energy capacity and we need to get more innovation into that and we need more of those supply chains in the U.S.
Nuclear, one of my favorites.
When was the last nuclear reactor that came online in the U.S.?
Right, you know, it was 20 years ago.
Right, you know, we just stopped, you know, a very capable industry.
So that's when we need more.
You know, we need to get much more efficient at delivering.
you know, the power networks.
And as you said, you know, moving 800 volt DC data centers, absolutely, you know, right, you know, and being able to eliminate so many conversion steps along the way because of historical, you know, reasons and, you know, standardization reasons.
Okay, we got to fix this, you know, an 800 volt DC, the revenge of Edison is upon us.
Yes, I was just going to say.
Yeah, I like to say.
So, you know, I do think that ends up being, we need better power conversion.
We have a couple of companies, one of those vertical GAN, I think will be a killer technology to be able to do 800 to 48 or 800 to 12 or even 800 to 5 in one conversion step.
right, which just gives you a lot more energy.
Solid state transformers?
Yeah, you know, all of those are solid state transformers, switches, et cetera, you know, rebuilding, you know, the power delivery network.
And I think that will, you know, have a lot of, you know, innovation, you know, sort of like, okay, power's cool again, right?
You know, forward.
And then you're going to need all of those in the cooling systems as well.
Right, you know, and so there'll be a bunch of innovations in that, you know, so, you know, turbines and refrigerants and, you know, all of a sudden they get to be, you know, exciting new technologies again.
So, you know, for all of us hardware guys, this, you know, is like renaissance in Brunner.
It's going to be, you know.
After 25 years, I had to go back and read about Cardinal's Law.
Cardinal efficiencies.
Oh, yeah, this is like good stuff.
How does that first law of thermodynamics work, I guess?
Yeah, yeah.
Physics, I think there was something after all.
You know, so it's going to be, you know, an exciting time that way.
You know, but, you know, we also say, you know, we need to change the physics, right?
Because CMOS hasn't improved, you know, the essential power per tera, you know, power per tera flop for the last five generations of NVIDIA chips, you know, flatline.
Not okay, right?
And that's where some of the things, you know, we have a superconducting company, Snowcat, right, which affords the opportunity to...
fundamentally change the physics, you know, a thousand times better power performance.
So I do think some of those innovations are upon us in the near future as well.
I've been a terrible time manager here for the past hour.
So, so many topics yet to cover, we didn't cover, but we got to cover at least one before we leave.
Okay.
Because now...
You're in charge.
Yes.
So I thought.
Now VMs are back.
How do you think about that?
The former CEO of VMware?
Well, you know, I...
I feel excluded from this conversation.
No, no, no.
Part of it was I was going to talk about networking as well.
But I don't think we want to talk about it.
But, you know, hey, I think the, you know, every innovation has always led to the next abstraction.
And I think many of these ideas, I would like to say, what's new in our world today?
Data structures, algorithms, and abstraction.
You know, we bring it back to those, you know, foundationals.
And, hey, you know, I think, you know, the virtual machine abstraction, you know, whether it's done from the infrastructure level or from the application level, you know, I think it's still a foundational abstraction model, you know, that deserves to always have a place in the compute hierarchy.
So, you know, so what do you think?
One of us go back and run VMware?
Yeah, that's what I tell.
I think I told Guto, it's time to go back and run VMware again.
Yeah, yeah.
And, you know, and I...
you think some of these abstractions, you know, but you also think about it and like one of the things we did at VMware, we, you know, essentially, you know, managed every aspect of the computing hierarchy.
Yeah.
Right, you know, and you manage the workload, you manage the network, you manage the storage, you know, system and how do you abstract those?
And I think in the AI context, right, we think about, you know, these agent swarms, well, who's going to manage the agents?
Who's going to create the security profiles around all of the agents?
Who's going to manage the performance of the agents?
You know, essentially every fundamental element of virtualization and management needs to get recreated in this next computing hierarchy.
And there's going to be lots of wonderful companies that you'll sort of break through for how those things will be, you know, done.
Okay.
Do you want to end it on networking?
Actually, I'd love to stay with the VMs for a second.
The most interesting thing for me is that we're now building VMs so that they're usable for agents as opposed to usable for humans.
And what we've seen in some of other companies is that the change is a lot, the form factor that changes a lot, how you market them, right?
You want the agent to make the pick for your VM offering.
It changes a lot how much complexity you can have.
It changes how quickly the startup should be.
Humans are a lot more patient than agents.
So if you have to rebuild VMware today, how would it look different?
Yeah.
And I do think...
VMware for agents.
Yeah.
And I do think there's, you know, does there need to be a VMware for agents for humans?
Right.
I see what you mean, yes.
Right.
You know, in the sense that, you know, I need humans...
Let's stick to the agents.
They're way more interesting than they are right now.
Yeah.
But, you know, I think it's an important point because I do think that there needs to be, you know, and I'll use, you know, Dario's constitution, right, or guardrails, right, or being able to set policies.
And I think then you really say, okay, every aspect of the virtual machine, as we know, we'll love it today, it is servicing agents, right?
As opposed to servicing hardware and humans, right?
So I think, you know, that becomes your fundamental design constraint on this side.
You know, how do I make agents great?
Right.
How can I make them secure?
How can I make them performant?
You know, you know, you know, how do I abstract them?
How can I migrate them?
All of those things, you know, have the new embodiments, you know, V motion for agents.
Right.
It's like every one of these will have the new embodiment here for them.
But then I also need to think about how can humans be setting the policies, describing the constitutions, you know, getting the dashboards, et cetera.
And I think that's sort of where the interesting thing is because you need to to do both of those, right?
Not just one of those, but the fundamental, you know, I'll say the hard design stuff like we did for, you know, virtual machines was associated abstracting the hardware in a performant way for applications or operating systems here.
It's, you know, abstracting the hardware and the operations for agents.
It's a fundamental thing that you're serving.
It was great discussing this.
Thank you so much, Pat, for coming in and we'll do it again.
Well, two of my favorite people anytime.
All right.
That was awesome.
Thank you.
Thank you.
Thanks again for listening and I'll see you in the next episode.
As a reminder, the content here is for informational purposes only.
should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.
Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast.
For more details, including a link to our investments, please see a16z.com forward slash disclosures.