# World Models: Spatial Intelligence for Robotics

**Podcast:** a16z Podcast
**Published:** 2026-09-13

## Transcript

Language models are these general horizontal engines for processing streams of discrete text, discrete tokens.
Those have had tons of applications from, you know, everything that we know and love today.
And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding, that can be used to generate, simulate, reconstruct worlds.
And if we can build these with the right generality, they should be applicable to tons of different industries, right?
From entertainment to VR to construction to robotics.
We live our lives in this physical built space all around us, and we need models to help us with these things as well.
Language models gave AI a way to work with words.
What happens when models can understand and simulate the physical world?
World Labs co-founder Justin Johnson joins Theo Jaffe and Sofia Puccini on MTS to discuss Atlas and the broader idea behind world models.
They explore how models could generate and reconstruct environments.
What separates them from video generation?
and applications across gaming, VFX, and robotics, including turning a few photos of a real space into a simulation for training a robot.
We're live with Justin Johnson, who is a co-founder of World Labs, which is a startup that builds spatial intelligence products.
And just today they announced Atlas, the world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D.
Model the world, move the camera, and simulate space and time.
So congrats on the launch.
Like the launch video.
Looks really cool.
Welcome, TempTS.
Yeah, thanks for having me.
Absolutely.
So tell us a little bit more about what this product is specifically, what you can do with it, what people will use it for.
Yeah, so this is not a product launch.
This is a model announcement.
So the products are coming later.
This is the base model.
It's going to be used to power our future products from World Labs.
But at its heart, it's a world model.
It does basically three different kinds of things.
Generation, reconstruction, and simulation.
Generation is you want to generate new worlds that don't exist.
Right.
Like I can start from a text prompt.
I can start from an image prompt.
Then I can direct the camera and sort of fly through that world and whatever.
Have the model generate a world that never existed before.
So that obviously has tons of applications across many things in creativity.
Right.
From VFX to gaming to all the kinds of use cases that people are using there.
The second major capability of Atlas is reconstruction.
Right.
There sometimes I don't want to generate a new world.
Sometimes I have an existing space.
in the real world that I like to model and bring into virtual reality or the virtual world in some way.
Right.
Right.
So there what Atlas can do is sparse reconstruction.
I can take just a few photos of a space, as few as one up to 100 or more.
And the more photos you give it, the more you can reconstruct that existing real world space accurately.
And the third thing we can do is simulation.
Right.
And this is going after more robotics.
Yeah.
Well, both VFX and robotics.
But there we want to be able to take an existing space.
recreate that space in 3D, then simulate how a robot might behave in that space.
And we show some examples of all these different use cases on the blog.
Yeah.
Is the ultimate vision for WorldLab something more like environment for teaching robotics how to work?
Or is it more like a metaverse that people will like...
spend time all the above.
Or is it just kind of all the above?
Yeah, I think it's all of the above.
The central thesis here is that there's a new there's a different kind of horizontal model that we can build called a world model, right?
Like language models are these general horizontal engines for processing streams of discrete tax discrete tokens.
Those have had tons of applications from, you know, everything that we know and love today.
And our thesis is that there exists another category of model called world models that should be based in visual understanding, should be based in physical understanding that can be used to generate, simulate, reconstruct worlds.
And if we can build these with the right generality.
they should be applicable to tons of different industries, right?
From entertainment to VR to construction to robotics.
You know, we live our lives in this physical built space all around us and we need models to help us with these things as well.
Right.
So the model is the input that would go to all of those industries, like the input intelligence.
I'm curious like how this has evolved and why Atlas is a big jump here.
So I would imagine like one of the difficulties with procedurally generating worlds is that like after a certain point, like I feel like, you know, With video models, like 10 seconds in, I feel like the output could start to get jumbled, especially if you have like a reference image.
And then it starts to take its own direction.
It starts to like hallucinate in its own way.
So how do you fix this thing of like at a certain horizon, things would start to get jumbled or distorted?
Yeah, that's a great question.
So here we built Atlas specifically around spatial control and the notion of a spatial context.
So when you give Atlas reference images, they aren't just images.
They're actually grounded in 3D space.
So you can stage those images in 3D and say this reference image should be here.
this reference image should be here.
This reference image should be here.
And when you generate a video from Atlas, you're not just giving it a text prompt, you also go in and steer the camera and say, I want the camera to fly this way, this way, this way, this way, with exact precise pixel control.
So then we show some examples where even up to a minute of video, we give it more reference images and sort of leave breadcrumbs of those images along the trail in 3D.
And then you get that direct tutorial control over the video that you're generating.
So the idea here is that even as you go to really long generations, it's not a slot machine.
You're generating and directing and really controlling what the model outputs.
Yeah.
So, yeah, we were just talking right before we went on about our friend Tyce from OpenAI did this project that went kind of viral that was like GTA, but it's San Francisco.
So it like, I think it was like a giant Gaussian splat of the whole city or something.
And you can like walk around in it.
You can steal cars and drive around.
You can fly around.
It's not very high resolution.
Definitely not as much as Atlas.
You think ultimately this is something you'll be able to do with Atlas?
Yeah, I think so.
And I think there's actually two different futures here of like the tech path that gets us there.
One is sort of the explicit 3D route that you mentioned, right?
Like the idea of generating a Gaussian splat reconstruction, then importing that explicit asset into a game engine or something else.
This is something Atlas can do, right?
Like it can be used to generate, you know, 3D Gaussian splat scenes.
And then you could import them and generate and build games around them.
Another direction where this could go is, you know, you skip the engine entirely and go directly for frame generation.
Right.
And I have the model directly outputting the frames without any explicit 3D in the mix anywhere.
And Atlas is built to do this as well.
So it's one model that both handles, you know, 2D when you need it, 3D when you need it, and can handle both modalities in a native way.
If you're somebody using this world model and you have all these inputs, like how would you...
How would you steer Atlas to do what you intended?
Like if you're like a film producer or video game developer, how would you work with it?
Yeah, we have some internal tooling for this, but it's going to take a little bit more iteration on the product side to figure out the right way to expose that.
But the internal tooling is actually pretty awesome, right?
You can basically take images, drag them into this interface.
You have this sort of spatial 3D canvas for all your input images.
Then they end up with this like proxy geometry in 3D space and you can steer your camera kind of with an FPS type control to like decide exactly where you want this thing to go.
And that's a pretty complicated user experience or user interface.
So I think we'll probably iterate on different ways to expose these capabilities in nice ways.
How does it differ from Marble, which is WorldLab's first product?
Yeah, so it's a totally rebuilt stack of models, right?
I think the central thesis is the same.
We want to build multimodal models that can input many different modalities, output many modalities.
This is something that Marble did as well.
But one way in which they differ is that Marble really put Gaussian splats really at the center of many things that it did.
So Marble could output videos, Marble could output 3D scenes, Marble could output meshes, but everything was kind of bottlenecking through a 3D Gaussian splat representation.
So whenever we output videos or images from Marble, you were always generating a Gaussian scene, then rendering those Gaussians back to images or videos.
And this is really nice, but we wanted to break that dependency.
So Atlas, you know, was built fully from the ground up to...
both rethink the architecture to make it more unified, more easy to scale, and to have that sort of that branch between 2D and 3D happen earlier, right?
So that when we're generating 2D outputs or video outputs from Atlas, you're not bottlenecking through a Gaussian splat scene anymore.
The model is directly giving you those beautiful 2D pixels.
And then only you lift them up to 3D only for the applications that you actually know that you need 3D.
So I guess at this stage, like, are we outgrowing Gaussians as a technology or like...
What is their role within all of this?
I think Gaussians are a tool just like anything else, and you've got to use the right tool for the job, right?
Like if you're working on something like VFX, then the ultimate output is a 2D image or a 2D image or 2D video, and maybe that's all you need.
But for other applications, especially real-time applications on embedded devices, like I think Gaussians and other explicit 3D representations still have their place.
I think there might be a world in the future where we're all streaming pixels directly off these models in real time from powerful server farms, but we're not quite there yet.
And even if we are, it's going to be pretty expensive.
So something like a Gaussian or a mesh is really useful for client-side rendering, especially on mobile or VR devices, and also really useful for integrating with existing workflows and pipelines.
I think we all believe in the future of technology, and we're excited about building these solutions that...
you know, give us whole new experiences.
But the reality is that it takes a lot longer than these, it takes a lot longer for these things to permeate than we expect, right?
So people in gaming and VFX and design and architecture, they have their existing workflows that work on 3D representations.
And, you know, instead of trying to switch them over all at once to a fully AI native thing, if you can get with AI into their workflows and meet them where they are, then that's another great use of splats and other 3D, and other explicit 3D representations.
Got it.
Yeah, one of my favorite video games ever is Microsoft Flight Simulator, which is you get in a plane and you fly around the world and it uses like a lot of, not quite real time, but it uses satellite data in order to like reconstruct a 3D model of the world, basically, in reasonably high fidelity, not incredibly high, reasonably high.
Do you think Alice will eventually be able to...
Like, what will it take to get Atlas from where it is now to able to dynamically generate content quickly enough to do what Flight Simulator does?
And like, is this even like the right tech tree for that?
Imagine like Google Maps in this world, like you could just traverse like any space basically.
Is it kind of overkill though?
I mean, I think it'll happen sooner than you expect, right?
And that'll happen in one of two ways.
Like there's one sort of, and actually, you know, Atlas was built to handle both of those tech trees.
One is the explicit 3D route.
Maybe we're generating a Gaussian scene or a 3D point cloud scene and then interpolating that on the client.
That's something Atlas can do because of its native 3D outputs.
The other version, like I said, is just directly streaming the frames.
And that's something that I think we'll be able to do pretty soon.
Okay.
I'm also really curious about reel-to-sim for robotics.
So, you know, taking like a reel video and then building a simulation.
So how is this useful and how do you see this possibly being, like helping evolve the field?
Yeah, so I think there's a couple different ways that this is useful.
One is for evaluation, right?
Like you definitely want to evaluate robotics in the final situation, in the final environment where they're actually going to end up, right?
If you're going to have a robot working in a particular factory environment with a particular type of machine in a particular environment, like you want to make sure the robot works exactly in that intended environment.
So there's a really useful role in simulation at that like last mile, that last mile part of robotics training and evaluation.
where we build a simulation that matches the final use case as accurately as possible.
Then even if you've got your big pre-trained robotics foundation model that knows about a lot of robotics actions in general, then you could either fine-tune that thing directly in that final simulation that matches the end application or even do evaluations to make sure that it's going to work.
So that gives you a different avenue than you could just collect a ton of demonstrations of a robot either working in that space or a person teleoperating the robot in that space.
But simulation gives you a different angle to simulate.
those environments.
Yeah, interesting.
So is there a world where like, you know, you have your like general pre-trained robot, but then you always need like a realtism of a specific environment?
And could that get us basically to like a working like general robot faster by scaling this?
Maybe.
I think there's a couple different tech trees that people are working on.
One is this notion of fully in-context learning, right?
Like maybe I've got a robotics foundation model, then I can kind of demonstrate the robot once how a task should be performed.
And that's enough for the robot to figure out, like stick that demonstration in the context.
And now the robot can figure out that task.
Another version is more this, you know, real to what we're calling real to sim to real.
There, maybe I've got my pre-trained robotics foundation model, but I want to adapt it to this particular environment, right?
Like this studio, like this table, like moving this microphone around.
Yes.
Then there's a version of that where you could come in here and take just a couple of casual videos with your phone or a couple of images with your phone, use Atlas to reconstruct the space, and now stage all kinds of robotics interactions in this studio space in particular.
And if you could lower the barrier to doing that, like literally you could do it in five minutes.
to like take five photos, upload it to Atlas, have a general simulation, maybe describe in natural language what you think, what kind of task you want the robot to do, have an agent go build, you know, build that simulation for you, fine tune your thing, like RL, fine tune your general purpose robotics foundation model.
And now you've got a robot maybe in the span of a couple minutes.
Yes.
That could come in and is perfectly adapted to this space.
You like a very fast onboarding.
Exactly.
Yeah.
That'd be great.
I'm kind of curious about All the way back in 2023, 2024, when the original version of Sora came out, opening eyes paper on it was called like video generation models as world simulators, which is seems like very similar to the vision of world labs.
And then, you know, they did Sora as that.
They did Sora 2 basically as like a TikTok clone that didn't seem that interested in.
the fundamental research direction of video models as world simulators.
It seemed like a consumer product instead.
And then they dropped Sora, and now they're not working on world models at all.
So why do you think they decided to drop it?
Well, I don't want to comment on OpenAI in particular.
I don't really know much about their internal strategy or discussions.
But our thesis is that world models are this general horizontal platform that can be used to go after a lot of different applications.
And there's going to be a ton of value unlocked there.
I think there is maybe a sense of a bit of innovators dilemma, right?
Like you're sitting on LLMs and you've got this as a really powerful thing and you're leading the pack there.
It doesn't necessarily make sense to try to invest in that next thing.
But, you know, for us, we think that there is another category of models out here.
There's a ton of opportunities out here and we want to be the ones to go and grab them.
Yeah.
Do you think that there's like, I'm curious about the mass like consumer applications of world models are the first ones.
I feel like.
VR or AR feels natural and then video games feels very natural.
I feel like there could be like, we could start shifting in a direction where everyone's like vibe coding video games and world models could be a part of that.
Whereas right now we're just like vibe coding websites.
We're already kind of doing that, right?
Yeah, I think we're moving towards that.
Do you see like world models being like that rapidly deployed?
so that everyone can kind of play around with them.
Yeah, definitely.
And that's what we want to do.
So we want to get this, that's kind of like fits into our thesis about these models as horizontal general things, right?
So if we want to expose these models to a lot of people, let them, you know, be consumable by humans and agents alike and let them interoperate with other tools, right?
So if we can provide a model that has all these general capabilities in simulation and reconstruction and generation in explicit 3D and 2D, and then we can have an agent, a coding agent go and stitch all these capabilities together.
then we can let people really quickly iterate on really cool experiences.
So I'm really excited to see once we get this into people's hands, what kinds of cool experiences people will build with it.
Right.
Yeah, I'm also curious, like, how you think they'll naturally work with LLMs.
So are we all going to basically be, like, adding another subscription to our AI stack?
And, like, do you think, like, agents are very easily able to, like, work with world models in theory?
Yeah, definitely.
I don't see why not, right?
Once you offer something on an API, like, you know, you get certain things in, certain things out, like, you can, they interoperate with other pieces of software.
Like, I think we've always wanted to build these things for both humans and agents to consume.
Yeah, I do wonder if, like, GTA 6 is just the last great piece of, like, human-written software before, like, all future video games are kind of generated on the fly with stuff like Alice.
I don't know.
I disagree with that.
I think there will always be, like...
human in the loop with, because video games are an art.
I'm not saying, I'm not saying there won't be humans in the loop, but I'm saying like, But like fully hand stitched to like video game?
Yeah, like, you know, you won't need to like, you know, physically place like millions of different assets in a game engine or anything.
Yeah, but this is just the evolution of technology in general, right?
As we see like, you know, from the early, like some of the early Disney animation films where they go and they hand paint every cell, like that was already a new type of technology that enabled new, a new way to make movies and creative experiences at that time.
Right.
Or some of the early Disney films figured out how to sync audio and video for the first time.
And that was a really transformative experience.
But, you know, new technologies come out and creative people always find really interesting and innovative ways to enable new experiences.
So moving from, you know, live, like live action, silent films to like animated, animated early Disney films to.
you know, the kind of computer generated films, like, you know, the Pixar era of CG, you know, into AI.
I view these as just new creative tools and there'll always be people, like creative people behind the wheel that will be interested in creating new experiences.
Yeah, I can imagine like a Pixar, but in 3D, like in like a 3D navigable space sort of thing emerging.
Like, I think this is going to be really good for creatives specifically.
Yeah, I think so.
Right.
And what we've wanted to build on all these things is this aspect of creative control.
We never wanted to be building these generative slot machines where you just like pull the thing and hope you get out a good generation.
Or you don't feel like that generation was yours or that you controlled it or that you knew what you were getting out.
Like we want to build these tools that have really deep control, right?
Like whether it's 3D control, whether it's text control, that you feel like you're a director in the thing guiding this model and deciding exactly what it outputs rather than just like the model spits something out and maybe I can stitch it together after the fact.
Yeah, totally.
How does Alice perform with like difficult physics stuff?
I want to do a simulation of a black hole or something.
Like, would it be able to say like, oh, this is like a scenario where like normal object collisions don't really apply.
You'll need to do like spaghettification or whatever.
Yeah, black hole is a good one.
I haven't tried that.
Maybe I'll try this afternoon and tweet it.
You should do that.
You should, you should.
I'm actually really curious to hear.
Yeah, my guess is it probably won't do something super exciting for black holes.
We'll see, we'll see.
Yeah, is the value...
Or what about just like hair or something?
Hair is pretty good.
Like I think one of the most amazing applications we got out of this thing that was actually a bit of a surprise to us how well it worked were these bullet time shots.
So we can do this thing or you can take like a couple of regular iPhones and stick them on tripods, just like three iPhones, take synchronized capture of some event happening.
And then you can use Atlas to do these bullet time reframing shots, like freeze frame fly around.
And this worked better than any of us were expecting.
It's completely insane.
So you have one of the one of our headliner examples.
There is a strawberry dropping into a bowl of oat milk.
And then we can like as soon as the strawberry hits the milk, the milk splashes up.
We freeze time and like go in there and fly all around it.
Like it's just incredible.
This works from just three input views.
Yeah.
I'm also curious about that, like added capability of being able to like adjust where the camera is.
Does that help?
Like what part does that help with?
I can imagine like that helps if you're trying to make like a movie or something.
But yeah.
how is that a useful capability?
Yeah, I think it's very useful as a, so I think there's a couple notions, places where 3D is useful.
One, like we already talked about the output side, like sometimes you want to have that explicit 3D output to plug into a workflow or a game engine or something like that.
But I think there's an underappreciated value of 3D on the input side as well, right?
Like even in a situation like a VFX shot or something like that, where the ultimate output is 2D.
it's useful for the model to offer these 3D control surfaces to let the human director figure out what kind of scene the model should output.
So in this case, we can do this.
There you can control precisely where does the camera move because I don't just want to write in text like pan in, pan out, truck left.
I want to be able to grab the camera and make it zoom exactly as I like.
And there's really no way to describe that explicitly in text.
You really want to be able to just grab the camera.
and make it do exactly what you wanted.
So Atlas has camera control as a native input type, right?
It's multimodal, and one of those modalities is cameras.
Yeah.
So if you think it wouldn't do a black hole correctly now, like what more would be required in order to get it to handle all kinds of edge cases in physics?
Oh, I mean, so I think there's a couple different regimes of physics.
So like anything that's like roughly Newtonian, like that was in the pre-training data.
So anything that's like roughly Newtonian that doesn't work at is probably just a quick SFT away.
If you want something that's sort of non-Newtonian, right, like black hole physics or like, you know, microscopic stuff, once you're down in the quantum mechanics realm, I think then you'd have to rethink a bit the architecture and the data side.
But the black hole example is really interesting.
I'll have to give this a try.
Yeah, yeah.
I would definitely be very curious to see, or like at a nanoscale.
Yeah, that's probably not going to work.
Yeah.
Not yet.
Not yet.
But one of our dreams we joked when starting the company is you want to build these 3D world models.
And how cool would it be to use a 3D world model to design a chip that we then use to train the next generation world model?
I was literally just thinking that.
Yeah, yeah, yeah.
Because we're talking about quantum stuff.
You have your like semiconductor.
Also, to me, what feels very natural is that like the natural modality for navigating these and even prompting the world models is just going to be VR, basically.
Yeah.
Because you can like, I guess, do WASD and then like move your mouse around to traverse a world.
But I think eventually you will have to basically like...
Well, not just eventually.
We've had Marvel in VR for like almost a year at this point.
That is a really transformative experience if you guys haven't tried it, right?
I haven't tried it.
Then you were like, really like stick on the headset and like I'm in this complete AI generated world that I created.
Like it's quite amazing.
Yes.
Yeah, yeah.
Yeah, absolutely.
Yeah, I can't wait to see all the amazing stuff that people are going to do with Atlas once it's fully out into the world.
Yeah.
Yeah.
Justin, thanks so much for coming on MTS.
Yeah, thanks for having me.
Absolutely.
This was awesome.
This was really awesome, yeah.
Thanks for listening to this episode of the A16Z Podcast.
If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family.
For more episodes, go to YouTube, Apple Podcasts, and Spotify.
Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com.
Thanks again for listening and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product.
This podcast has been produced by a third party and may include paid promotional advertisements, other company references, and individuals unaffiliated with A16Z.
Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates.
Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.