# Local AI Models: Building Resilient Business Infrastructure

**Podcast:** The Startup Ideas Podcast
**Published:** 2026-06-13

## Transcript

I had my entire weekend planned out.
I was going to lock in and use the most powerful AI model on the planet, Fable 5, to build this crazy idea I've been sitting on.
Then Friday at 5.21pm, the US government sent Anthropic a letter.
And by Friday night, the model was gone.
Disabled.
For everyone.
No warning, no appeal.
And I sat there thinking about how fragile this whole...
So this weekend, I'm not building with any frontier models, not none.
And this is the episode I needed to make.
By the end of this episode, you're going to understand what local models are, why they suddenly matter more than they did a week ago.
exactly which ones to use, what hardware you need, and a few startup ideas that only exist because intelligence now runs on your desk for free.
I think it's opened up a bunch of money-making opportunities that I'm going to share by the end of this episode.
Let's get into it.
So let me paint the picture of what actually happened because the lesson is bigger than just this one model ban.
So frontier models are incredible.
I'll be the first to say that.
Nobody's arguing that.
But they all share the same weakness.
You don't own them.
You rent access.
And rented access could be revoked at any time by a government, by a policy change, by a pricing change.
They could just make it so expensive that you can't access it.
By the company deciding your use case, violating a term you didn't read.
We just watched this happen in real time.
The single most powerful model on earth is gone overnight.
And I want to just be clear that I'm not anti-cloud.
I use these cloud models every day.
And the cloud models are going to be the strongest.
They're going to be better than local models, just in terms of you're getting the best possible stuff.
They're the smartest tool available.
But what it's taught me is that you do need to own a part of your stack.
You need a layer that nobody can take away from you.
And the way I think about it is like electricity.
Most of the time you're happy being on the grid, right?
It's cheaper.
It's easier.
Someone else maintains it.
But the people who are truly resilient have a generator in the garage.
You know, a hurricane comes and lights go out.
Well, they got this generator that continues going and they can actually use their stuff.
Local models are basically that generator for you.
And I know what a lot of people are going to say in the comments are going to be like, well, local models aren't good at all.
And it's just not that true anymore.
I think the switch probably happened about six months ago.
Two years ago, running a model on your laptop was literally garbage.
Maybe a year ago too.
But today, a model that runs on a gaming GPU or a decent Mac is good enough for about, I would say, 80% of what most people use things like ChatGPT or Cloud for.
The gap between free and local and expensive cloud closed faster than I think a lot of people expected, including myself.
So let's actually talk about what a local model is.
And I want to make it...
Dead simple.
You know how I am on this channel and this podcast.
I'd like to just dumb it down for myself and for you because I don't want to scare people off.
A local model is an AI model that runs entirely on your own computer.
You don't need internet, you don't need an API key, and you don't need per token cost.
No company is watching what you do.
You just download.
the model file once.
And from that point on, it's yours.
It runs on your machine the same way a video game or a photo editor might run on your machine.
And that's really it.
That's the whole concept.
We don't need to overcomplicate it.
Basically, the intelligence lives on your hardware instead of someone else's.
And you get three main things that you don't get with cloud models.
The first thing you get is privacy.
Your data never leaves your machine.
And this isn't nice for you just personally.
It's an entire unlock for selling to a bunch of different industries that you might want to sell to, like healthcare or legal or finance.
Industries that legally cannot send their data to a third-party API.
And there's actually a ton of those industries.
We're going to talk about more of that when we get into the startup ideas.
So let's put a hold on that.
The second main point is you get zero marginal cost.
So after you've got the hardware, and of course you do need to spend money on hardware, and hardware is getting more and more expensive.
But after you've got the hardware, every query is free.
It's unlimited.
And you can run a model 24 hours a day for a month.
And your bill is just going to be the electricity.
That does really change the math on an entire category of products and it opens up a lot.
The third thing is nobody can turn it off.
The model on your drive works whether or not the company that made it even exists.
Whether a government likes it or not doesn't matter.
Whether or not your internet is up.
It works on an airplane.
It works in a bunker.
It just works.
So yes, you get some main benefits, but with everything in life, there's pros and there's cons.
So let's talk about what the trade-offs are.
Because I don't really want to sell you a fantasy.
I'm not here to sell you a fantasy.
I'm here to tell you what are the pros, what are the cons, and what I think is interesting about it.
The trade-off is that local models are generally not as smart as the absolute frontier models.
Open models can match the cloud, but they need serious, serious hardware.
You'll see people on X and they are doing insane things with local.
A lot of the times they're spending $5,000, $10,000, $15,000, $20,000 on machines.
The ones that run on a normal laptop are a notch below the best cloud models.
But the way I'm starting to think about it and reframing it is you don't need frontier intelligence for most tasks.
You need good enough intelligence that's private, free, and always on.
And then you've got to match the right model to the right job.
And that's becoming a whole new skill set.
And we're going to get to that.
So how do we get good at local models, which is something that I'm spending my weekend trying to figure out?
and sharing everything in real time.
This is really the meat of this episode.
If you really want to get good at this, and not just nod along and watching YouTube videos and podcasts, here's the order I'd learn it in.
The first is start with runtime.
Everyone gets this backwards.
They go hunting for the perfect model before they can even run one.
That's the wrong order.
The first thing you download is the runtime.
the program that actually runs models on your machines.
There's two main names to know, Ollama and LMStudio.
Ollama is usually the favorite of a lot of my developer friends because it runs from the command line.
It's relatively simple because it's one command and then it runs the model.
But LMStudio is the one I'd start non-technical people on because it has a real interface.
It's got a model browser.
You click and it runs.
And there's no terminal and those things are scary.
This is sort of the part that a lot of people overcomplicate it.
Just download one of these first, whichever one seems to resonate with you more.
And you'll have a model running in 10, 15, 20 minutes.
The second thing is you're going to want to match the model to your hardware.
A model's size is measured in billions of parameters.
You'll see numbers like 4 billion, 12 billion, 27 billion, 70 billion.
Bigger basically means smarter, but bigger also means more memory to run.
The single most useful thing to understand in this entire episode is the rough mapping of model size to hardware.
A 4 billion model runs on basically anything.
an 8-gigabyte laptop, even a lot of phones.
A 12 billion model is the sweet spot for a machine with 16 gigabytes of RAM.
This is where most people should live.
A 27 to 35 billion model needs a really good Mac with 30 gigabytes or more or a dedicated GPU.
This is where it starts feeling genuinely capable, in my experience.
A $7 billion and up model needs serious hardware.
A maxed out Mac Studio or a dedicated box like an NVIDIA DGX Spark with a 128 gigabyte unified memory.
The DGX Spark is interesting and I've talked about it on this podcast before because it's purposely built for exactly this.
128 gigabyte of memory, decides to stay on 24-7.
It runs Linux and it's really becoming the default for AI box on your desk for people who are serious.
I'm not affiliated with NVIDIA, just what I'm noticing in the industry.
You run your model on it, you leave it running and you connect to it from your phone.
So your desk becomes this almost mini, at least that way I see it as a mini data center.
The third main thing to know is knowing which model for which job.
There's obviously a bunch of models and I don't have enough time to cover all of them, but I'll give you the four main ones that you need to know about.
Quen 3 and the new 3.6 series.
The best all-around choice, I think, for most people, it's Alibaba's open model family.
It's quite strong at coding, strong at multilingual, it's clean commercial license.
They've got a 27 billion and a 35 billion versions and it feels like it punches above its weight.
It outperforms previous generation models four times their size.
If you only learn one, this is probably the one to learn.
But that's one of them.
The other one is DeepSeek.
You've probably heard of DeepSeek.
This is quite good at hard thinking and coding problems.
But heads up, the reasoning models take 10 to 30 seconds to think before they answer.
And that's normal.
If you install DeepSeek and you're like, why is it taking so long?
That's just usually what I've seen.
It takes about 10 to 30 seconds.
The third is Gemma.
And this is Google's open model.
And if I was Google right now, I would be launching a new version of Gemma right now and just taking advantage of this moment.
This one runs remarkably small.
There's actually a version that fits in 16 gigabytes of RAM.
And that's the one that can fit on your phone.
It's beautiful, clean writing.
The fact that Google gives this away for free is actually crazy.
And I wouldn't be surprised if Google double downs on this in the future.
Then there's Llama by Meta.
It's really become very important in the whole open ecosystem.
It's got a huge community, a ton of fine tunes.
It's got a lot of tutorials that you can go and check out.
It runs almost anywhere.
So when in doubt, there's probably a llama for your situation.
The fourth main point that you should learn around local models is what's called quantization.
This no one really talks about and it's a really important trick with respect to local models.
And quantization is this concept of shrinking a model so it runs on weaker hardware with barely any loss in quality.
The analogy I think of for this is a raw model is like a uncompressed photo.
Quantization is like saving a high quality JPEG.
It's a lot smaller and your eye really can tell the difference.
When you're downloading models, you'll see labels like Q4 or Q5.
Quantization is like, that's the compression level.
That's the quantization compression level.
And Q4 roughly halves the memory a model needs with pretty minimal quality loss.
And this is how a model that supposedly needs a server ends up running smoothly on your laptop.
So understanding this concept is really key.
And, you know, it's key because it's the thing that makes your...
Hardware suddenly do twice as much.
The fifth main point is you're going to want to connect to your agent.
So running a model and chatting with it is cool, but the real unlock is pointing an agent at your local model.
So you can use something like Hermes to do that.
I've covered Hermes.
I think last week I did an episode on Hermes desktop app.
You can go check that out.
Hermes is the most used agent in the world right now, I would say.
It's definitely gaining the most amount of hype and buzz.
And it's actually built specifically to run locally and never stop.
You point a Hermes profile at your local model and now you have an agent that runs free, runs offline, remembers everything, writes its own skills, and you can message it over your messaging app of choice like Telegram or whatever.
while the heavy work runs on the box of your desk.
So super cool.
Again, I have that episode that I did last week that I'll include in the description if people want to watch it and learn more about agent profiles and pointing it to local models.
So those are the key points I would say around what do I need to know about local models.
That helps you get up and running.
But how do we take it to the next level?
How do we separate the pros from the tourist?
One is the context window is your real constraint locally.
So cloud models hand you a giant context window for free.
That's the way to think about it.
Local models make you pay for it in memory.
So the bigger the context, the more RAM it eats.
So keep your sessions tight, super tight, and don't dump.
your entire life into one thread or your machine is going to choke and you're going to be like, local models aren't very good.
You're going to want to give your local model tools.
So a small local model with web search, file access, the ability to run code beats a giant model with none.
The capability gap closes fast when you wire up the right tools.
So think about it as the model is the engine and the tools are the wheels.
Common thing that happens with local models is sometimes it forgets your tools.
I don't know if other people have noticed this.
I'm learning in real time how to get the most of it, how it doesn't forget.
But just know that that is something that is a quirk that as of recording this, June 2026, that happens sometimes.
Remember that privacy is the killer feature here.
So everything is running offline.
Your data is not leaving the machine.
I'll talk about that actually more with the startup ideas and how you can leverage that.
The last thing I'll say about concepts that separate the pros from the tourists.
It's actually super helpful to run a small local model versus a frontier cloud model side by side for a week because that actually helps you build the instinct.
I think it's the fastest way to build the instinct actually.
And you'll be shocked with how often the free local model is good enough.
So you're going to see yourself stop reaching for the expensive option for things a 12 billion handles fine.
And that instinct, knowing what to run, is the skill that we're trying to learn here.
This whole Fable 5 moment of being banned and stuff like that, it's just a wake-up call for us to learn how to do local models.
And that's probably why you're here listening to me talk about it today.
So I wanted to give you some startup ideas.
I mean, after all, this is the Startup Ideas podcast.
I'm here not only to clarify how you...
How you learn how to use AI and be practical.
But I also am here for helping you get your creative juices flowing around startup ideas that only exist for a certain reason.
And there's some startup ideas that only exist now because local models exist.
And because a lot of people, I mean this is mainstream news, a lot of people are seeing like, hey, these cloud models could get banned.
So there's going to be a huge amount of demand.
in my opinion, for local models over the next few years.
So one startup idea I wanted to give you is on-device AI for regulated industries.
So this is a big one.
We kind of talked about it earlier, but healthcare, legal, finance, they have money, they have problems AI can solve, but they legally cannot send their data to a cloud API.
So a product where the model runs entirely on the customer's device, the data never leaves the building.
that opens a market that the cloud-based competitors can't enter right now.
So that privacy can train your remote and you just start selling to these types of people.
The second startup idea is you basically sell it as the data, your data never leaves version of existing AI tools.
So go pick any...
popular cloud AI product, note takers, meeting summaries, document analyzers, and then you just build local versions of those products.
It's the same product, but the pitch is basically nothing you give us touches the internet.
And you slap that on to the main value proposition of the landing page.
You do it for lawyers, you do it for doctors, therapists, and anyone handling sensitive documents.
That is the sentence that might help close the deal.
Third startup idea, the air-gapped agent for sensitive operations.
Some businesses can't be online at all for security reasons.
Defense contractors, certain financial operations, anyone paranoid about leaks?
You do an agent setup that runs fully offline on local hardware and they're going to have...
So it's not just the startup idea, number one, is just regulated industries.
But startup ideas, number three, is really around leakages and sensitive operations.
So you might have not such a sensitive industry, but they have a sensitive operation.
That's that niche.
The last idea I have for you is offline AI for places with no internet.
So ships, planes, rural clinics, field operations, disaster zones.
Useful agents that work with zero internet is a product the entire cloud industry simply just can't serve.
And then the last idea I'll give you is...
Resilience as a service.
So after this weekend, every serious company is going to be asking, what happens to our AI workflows if our provider gets cut off?
And you just sell the answer.
So it's basically a fallback layer that kicks in when cloud models disappears.
So you're selling insurance against exactly what happened with the Fable 5 banning.
is this.
This weekend, for me, was supposed to be about building with the most powerful model on the planet.
But instead it became about something more durable.
The lesson isn't that cloud is bad and local is good.
That's not the case.
The lesson is don't build your entire life on something that can disappear with a single letter.
Own a part of your stack.
Have the generator in the garage.
Local models are the insurance.
And this is the weekend I finally bought the policy.
And when you play with these local models, you're going to learn that, yes, they're not perfect.
Yes, they're not the most powerful model on the planet.
But for 60%, 70%, 80% of routine tasks, they're actually quite good.
And there's a huge range of those use cases.
Over the next few days, I encourage you to play with these.
Don't just watch this or listen to this and nod.
Download Olama or LM Studio, pull Quinn 3, run it, point Hermes at it, pick a real task and force yourself to do it entirely local.
And that's really how all the stuff clicks.
And once you actually play with it, you get your hands dirty.
You'll understand a little bit more of what I'm saying.
And so the next time something gets banned or something gets priced out of oblivion, you can still run your business.
You can still ship your ideas.
You can still do things.
And the best case scenario is you have cloud models doing XYZ and local models doing ABC.
If this was interesting to you, you learn a thing or two, Do yourself a favor, actually.
I was going to say do me a favor, but do a like, a comment, and subscribe.
That just means more of this stuff is going to appear in your feed and also tells me I should continue doing this and sharing what I'm learning in real time.
I hope you built something cool.
I hope you learned a thing or two.
I'm rooting for you.
Now go build something today that nobody could turn off and I'll see you.
in the next one.
Take care and have a creative day.