# Cloud RAG Economics: Borrow Plumbing, Keep Chunking

**Podcast:** The CTO Advisor
**Published:** 2026-08-25

## Transcript

All right.
Keith Townsend here.
I am launching or I've launched a new podcast.
We're the fifth podcast in.
And I guess this is how you properly launch a podcast by taking a popular podcast, embedding the new podcast into an episode and then giving you the link.
Search for it on your podcatcher.
Layer to see dot com.
Labs podcast.
This is basically the notes or the technical notes from the labs.layer2c that we've been running the past couple of months.
Converted into a podcast for you super geeks.
Again, if you're not really into AI for cloud from a technical layer, maybe not for you.
If you don't like an AI voice, you probably should stop now and wait until the next published podcast on the CTO Advisor.
Hope you enjoy it.
Subscribe, like it, et cetera.
Drop a comment.
Send me an email.
Keith at thectoadvisor.com.
Let me know what you think.
This is Layer 2C Labs, the podcast.
Episode 1, Lab 1.
Borrow the Vendor's Plumbing, Not Its Judgment.
Written by Keith Townsend.
Narrated by an AI because Keith doesn't read from scripts.
And this voice isn't pretending to be him.
Everything from here forward is his words.
And the title is The Whole Verdict.
The too long didn't read.
The managed data plane is cheap and fast, and the cost it hides is the chunking judgment it takes from you.
Borrow the plumbing.
Keep your own chunking.
Now, if you're listening to a lab podcast, you're probably one of two people.
Either you're an enterprise architect trying to figure out which parts of the AI stack you keep and which parts you hand to a cloud provider.
or you've already made that call and you want to check your work.
Hopefully, by the end of this episode, I satisfy both of you.
Here's the build.
A retrieval augmented generation pipeline.
RAG for short.
Over 13 years of my own transcripts.
Roughly 237,000 segments.
5,815 videos.
2013 to 2026.
I sent the data plane to AWS, Titan embeddings into an S3 vectors index, and I kept the compute at home on an NVIDIA DGX Spark sitting below the cloud's managed abstraction, validated on a 5,000-segment subset first, then ran the full corpus.
Not a sample.
The full 13 years.
And I watched where the decisions landed across the 4 plus 1 stack.
So what did it cost?
This is the part everybody asks about, and it turned out to be the boring part.
Embedding and indexing the entire corpus.
84 cents.
One time.
About 41 minutes, 337,000 segments an hour.
Zero failures.
Keeping the index warm is about 7 cents a month.
And the entire generation pass.
The whole 13 years of analysis would have cost about 20 cents at cloud rates.
20 cents.
Retrieval wasn't the constraint either.
From the Spark, I saw P50 at 280 milliseconds, P95 at 505, and most of that was my own network path.
The S3 Vector service itself sits around 100 milliseconds.
Cheap.
Fast.
Better than I expected.
If the story ended there, this would be a pricing note, not a lab.
It doesn't end there.
So what did the managed path actually cost me?
What the managed path actually cost me was chunking.
Here's what happened.
The managed service, bedrock knowledge bases, looked like the product.
When I went to stand it up, the control plane was gated.
It would not create through the API, even for an administrator account.
So I rebuilt the same data plane on the primitives underneath it.
Titan embeddings, an S3 vectors index, a query.
It worked.
And that rebuild exposed the thing I want you to take away from this episode.
The managed wrapper re-chunks your data on its own strategy.
And it turns on parsing you did not ask for.
That would have broken label-filtered retrieval outright.
Bring your own kept one segment as one vector with its labels attached.
And that's what made hybrid retrieval work.
Semantic similarity plus metadata filtering in one call.
Chunking determines what you can retrieve.
That decision is yours.
Don't seed it.
And here's the kind of funny part.
The ruling on AWS runs opposite to what you'd expect from a story about a gated console.
AWS's strength is the composable primitives underneath the managed wrapper, not the wrapper.
The wrapper is convenience.
The primitives are the capability.
So when you're scoring AWS, score the primitives.
Again, borrow the plumbing.
Keep your own chunking.
Now, you may say, Keith, you didn't keep the box busy.
Of course, the cloud won.
Utilization is the whole on-prem case.
Fair.
I ran that math.
Peg the spark at 100% for three years, and it generates roughly 1.9 billion tokens.
That spreads the $4,699 across enough tokens to get you to about $2.50 per million, plus around 40 cents in power.
Call it $2.90 per million.
Fully utilized.
The same model on Bedrock?
About 40 cents per million.
And it isn't just cost.
The box holds 20 to 23 tokens a second on this model.
Bedrock runs it at 136 to 214.
That's 7 to 9x faster for less money.
Utilization is the wrong lever.
The box is memory bandwidth bound.
It can't produce enough tokens an hour for the capex to ever spread thin enough to win.
And to hold 70 to 90% utilization, you'd need a constant fire hose of batch work feeding a box that loses even when it's fed.
You can't utile your way past a throughput ceiling.
Does that make the spark a failure?
No.
And I want to be fair to the box because it did real work.
It built and ran this entire pipeline end-to-end.
Local generation used Gemma 426B, a mixture of experts model with about 4 billion active parameters.
Quick model class note for you hardware folks.
That mixture of experts model beat a dense 8B on speed and on capability, on the same box, because single stream decode is set by active parameters, not total.
So place the box by its authority, not its benchmarks.
The spark keeps real local execution authority.
Its place is development and portability, not commodity inference.
And the architecture itself passed, which is the result the economics can hide from you.
The 4 plus 1 seams fell exactly where the model predicted.
Data plane in the public cloud.
Reasoning at the edge.
And the traffic across that seam was never the bottleneck.
Retrieved context coming down.
Queries going up.
Never the problem.
The constraint lived inside layer two in local generation throughput.
The designed partition holds.
That's not a diagram claim anymore.
That's a bench claim.
One more finding, and honestly, it's the one that set up this entire series.
The reasoning plane that yields a true answer, layer 2C in my 4 plus 1 model, is the gates, the judges, and the reproducibility around the model.
It is not the weights.
Generating LLM findings is free and easy.
Generating true ones is not.
It takes a quantitative gate, cross-run reproducibility, and a strong judge.
And judge strength dominates.
A mid-tier judge will rubber stamp a confidently wrong pattern that a strong judge refutes.
The fix wasn't the most expensive judge.
A cheap, strict judge plus one frontier judge, trusted where they agree, did the work.
Most of my naive discoveries did not survive that gate.
So even when you seed the model, you still own the harder half.
The lab also caught me being wrong, and I'm going to keep telling you when that happens.
My first note said, a frontier model wasn't available on one cloud.
It was.
I'd failed to access it, which is a different problem.
I corrected the record instead of shipping the mistake.
That's the point of a lab.
It tells you you're wrong before a reader has to find it.
Honest Bounds.
because a verdict is only as good as its edges.
This lab did not prove on-prem inference never pays.
It proved commodity base model generation on this box for this workload loses to the cloud on cost and speed.
It did not settle fine-tuned model economics because commodity per token pricing only exists for base models.
The moment you need a custom model, AWS...
stops selling you tokens and charges you a floor.
Lab 2 ran that regime, and spoiler, it reversed this verdict.
It did not prove managed rag is bad.
It proved the wrapper can hide control points chunking above all.
And it did not test the frontier.
An 8B-class model is a cost-structure vehicle, not a frontier contender.
The disclosure, and this one's short.
This lab was self-funded.
No vendor paid for this answer.
The full lab, every number, the authority scoring, and the raw detail are at labs.layer2c.com.
I'd love to hear what you think, especially if you've hit the chunking wall yourself.
And the question this lab left standing is where the series goes next.
The AWS native data plane was cheap and did hybrid retrieval in one call.
So where does a specialist data platform actually earn its keep over S3 vectors?
Latency?
Portability?
Retained governance?
Now I've got a measured floor to test that against.
Until then, talk to you on the next lab.
Borrow the plumbing.
Keep your own chunking.