# Database Branching: GitOps for Data Environments

**Podcast:** Thoughtworks Technology Podcast
**Published:** 2026-06-11

## Transcript

Hello, everybody, and welcome to another edition of the ThoughtWorks Technology Podcast.
My name is Ken McGrath.
I'm one of your regular hosts.
I'm thrilled today to be joined by both a ThoughtWorker and an external guest.
So I'll start with Kevin Hartman from Databricks.
Please go ahead and introduce yourself.
Thank you, Ken.
Thanks for having me.
So Kevin Hartman, working at Databricks as a partner solutions leader.
going to Databricks and really fleshing out my career in data and AI, I was a software developer craftsperson.
So I have a long career in experiencing the transition from Waterfall to Agile, live through those moments.
So I'm really excited to be talking with you today about how those moments now could be revisited with some really interesting changes in the way technology has come forward.
And Cam Kasher.
Hey, Ken.
Good to be here today.
Yeah, I'm Cam Kasher.
I am coming up on my seven-year ThoughtWorks anniversary in a couple weeks.
I've spent most of my time, or about split, being software engineering, data engineering.
So the topic today touches on both in a certain way, and it's pretty cool to see the cross-function of it as we get into this cool Databricks tool we've been exploring.
Cool.
So speaking of the topic today, Kevin and Cam have worked together on a blog post and a couple talks called Eliminating the Barrier Between Analytic and Operational Data Environments.
And as a longtime continuous delivery person, it really kind of struck a nerve and we were like, hey, we need to get them on here.
If we could, Kevin, I guess let's start with you.
A summary of what's the problem space you're trying to solve here, and what are you trying to do for people?
Yeah, that's a great question.
So the problem space is addressing the monolith in the room that was always rigid and never really very much adaptable in the way that Git innovated, the way we treat code, in that you really couldn't branch a database before.
to mirror what you do in a GitOps sort of environment.
And there have been workarounds.
We've addressed them in ways that are both creative and also somewhat expensive in some ways.
But it's now, I think, a really great opportunity to revisit some of those practices with what we have now in latest innovations to change how we treat the database and really align it even more tightly.
with the dev lifecycle.
So, you know, even in continuous delivery, you know, we talk about that you're going to constantly be pushing to production and what have you.
But the truth was, but especially when it came to the database, it was, you know, filing the ticket with that group or what have you.
I remember visiting a client when they were like, oh, these people want to push to production really fast, but we need to test the changes and secure it.
And how does this make that different?
How does this help alleviate that pain, what you're doing here?
Yeah, so.
Instead of thinking about this as multiple environments in which I managed that and replicated and set up the pipelines to make sure I had some level of quality data in which to test against, I could change that and say, no, I don't actually need to do that.
I don't have to provision a separate environment and keep those things in sync.
I can actually just fork directly from...
my database, treat my database like your main, called prod is main, right?
If you're doing the Git analogy.
And when I do a branch, when I cut a branch, I could do lots of interesting things on that.
One is I could first align that to my feature development.
I cut a branch from Git, I cut a branch from my database, and I can write real live integration tests as part of my My dev loop.
So TDD becomes a lot more real to the practitioner in that I don't have to put mocks in place.
I can actually create real BDD style behavior testing in my TDD testing.
I can also think about incorporating some of those steps and starting to shift the agile QA into a true.
pair programming sort of model again, right?
I don't know how many of us still do pair programming.
I know that a lot of it's in some organizations, it's controversial.
But, you know, if you think about what the intent is, and that is to produce and create quality code artifacts up front, so you don't have those risks downstream, this really allows that pattern to come to fruition.
Cam, as a developer, how does this change your behavior when you're thinking about code and with databases and the backends and so forth?
Yeah, that's a good question.
When I started learning more about Lakebase, it kind of touched on two points in my mind.
One was, what constraints is this sort of touching on and allowing to now happen from a developer standpoint?
Also, like, what are the new unlocks in a way that you might not have even considered when you're able to branch your database, essentially.
And, you know, having been on a lot of development teams where we've had to align on whether it's trunk-based development or having, you know, a development branch staging branch production, it brings that conversation to database development, which hasn't really been a consideration in as long as I've been working in tech.
So from my perspective, it just sort of unlocks this whole new way of thinking.
And that kind of all played into this sort of proof of concept that I tried to build off of and create to learn more about Lake Base, where I took...
a tool that I'd already worked with in the past called Backstage from Spotify, which is an internal developer portal, and tried to basically create a use case for how Lakebase could optimize it in whatever way.
And one of the really interesting ideas was how traditionally OLTP and OLAP had been you know, different storage layouts, different compute profiles, things like that.
And it really allowed you to bridge the gap and wire them together when you can branch and sort of point these two different sort of storage and database layers together and get certain insights that you wouldn't have thought otherwise.
Like even touching on a FinOps perspective of, you know, how much are you spending on having different databases and branches spun up?
So it seems like it is a bit of a paradigm shift in a way.
I'd love to see this really working with a client here soon from a consultant standpoint as a thought worker.
So really interested to bring in some of this information.
and have some real-life use cases to speak to as well.
So one of the things that you talk about in the article is this concept of developing against fiction.
And you take that further to say the door attacks.
What does that mean in this context?
Yeah, that's a great question.
So working against fiction is really this idea of working against mock data for your database testing.
And when you're working with mock data, you might not necessarily have the correct production shape or the correct values to test off of.
So you're ultimately testing against fictional pieces of data.
And when you introduce a tool like LakeBase, you can have branching swap mocks for live actual production shape branches where you can have the same shape, the same constraints, and the same data semantics.
When you tie this back into the DORA metrics, you could consider something like a change failure rate is dropping because we're able to use real data testing to catch what mocks might miss.
So you kind of have this idea where you don't necessarily need to develop against fiction, which is just a fancy way to phrase that.
Agents are writing code now too, right?
And so...
There's some speed there.
And we've always just kind of pretended that we're agile.
There's a quote in there where you talked about, we stopped talking about the part that was still waterfall.
Because the database stuff was still waterfall, but we're pretending everything's agile.
But now, especially, we have agents cranking out code at hyperspeed faster than we can review it, et cetera.
Does this help there at all?
What's your thoughts on that?
Absolutely.
If you think about what...
you do and don't want agents to do is you don't necessarily want them to make a change into production data directly, right?
It's like one of the things.
You also need to put other guardrails around what they do.
And frankly, agents need to follow and leverage the same practices that we do.
And unfortunately, those practices are built around that whole concept of creating a ticket, right, for a database change or something like that.
Well, the agent's not going to wait around for a ticket.
I mean, that's just so anti-agile already that now we're pushing that semantic onto an agent now to repeat that.
It's especially more important now for an agent because we need to actually adapt and have.
the ability for agents to do their work, to progress, to do experimentation, to discover what the right pathway is.
And you're not going to discover the right pathway unless you actually can test something against what is at least as close to live as possible.
Right.
And the branching paradigm of your database gives the agent to do that, the flexibility.
And it could be destructive.
So as soon as I create a branch from a database, I could do whatever I want with it.
I can apply my migration scripts to it or I can change, alter, whatever I want because it's something I can take and throw away later.
And that applies for the human in the loop as well or just the human doing it.
So getting back to the agent question, I think, unfortunately, if you don't have the...
prescription down as to like what to do and how to how i should process this how i should like take the branch cut from both places and do all these things an agent is just going to do something that it knows how to do as quickly as possible and it probably is not ideal right so an agent needs guardrails a framework per se uh to operate against and this is something else that you know we've been working on as well and that is uh an app dev toolkit, a kit that has those semantics baked in so that when I do the workflows and orchestrations as to in a coding loop, I have something to follow.
So it's especially important for an agent to have this information to truly leverage it.
And then I have to address for any of our longtime listeners.
What for me is an elephant in the room and probably not for others is that if you look at like ThoughtWorks, that was simple defaults, what we call them and stuff.
In fact, I did a talk on this.
I had to look it up.
It was 12 years ago that said that was titled branches, a long four letter word.
You know, we've we've we've tried to avoid long live branches, but those were feature branches that were different.
And as I understand it, this is what this is doing is giving.
the development team choices.
Can you talk a little bit?
I mean, is this in the face of main base development?
Is it, do they work together?
Is this an alternative?
You know, what is, how, when you say branch, do you mean something that just lives forever or what do you mean by that?
No, personally, it's more of a variable.
So I don't branch forever and keep branching.
I need to do a code merge in the same vein.
I need to also do, I need to throw away that branch and do a migration back to main or back to prod.
So features, if I'm pulling a feature, you can think about this in terms of how we've operated around scrum, two-week sprint cycles, release cut maybe at the end of a couple of sprints.
I accumulate all these things.
And this pattern could follow, right?
It certainly does move up any risks.
that you might encounter upstream.
So you discover them quickly when we have reintegration tests up front.
But that pattern and the way that we operate, we can also, you know, organizations do Kanban too.
And it's sometimes harder for organizations to do Kanban.
Well, this actually makes it easier for more organizations to adopt Kanban than ever before, right?
Because I'm actually doing things with more immediacy.
And I'm also not waiting for a test cycle downstream when now I've lost context on what that thing was.
And now a QA developer is asking me to go and fix something like, okay, what do I do?
What do I do again?
All those things are now live in the moment.
And I can really facilitate instant releases if I so choose.
So it's really up to the architect and the patterns that are in play at the organization to really decide.
And this really opens up that flexibility.
So it occurs to me that we've been talking for 10 or so minutes about the problem we're trying to solve without really describing the solution.
So what is it that Lake Base is doing here that's different that enables all of this?
So I guess, yeah, going back to the POC that I did, I wanted to basically...
look into all these claims that I was seeing around Lake Base and see if I could replicate them, get any sort of metric or data point on it and basically prove it out like a proof of concept.
So, you know, I was able to test out branching, see how quick it was, see how much lead time it could realistically drop for teams and determine that it is really applicable and can happen immediately.
like I was saying earlier, just unlock a lot of efficiencies and optimizations for teams.
Another thing that I determined was not only just branching, but point in time recovery is another option here, which is essentially just branching at a given time where normally you just branch at the exact moment or now, in other words.
So what branching at...
point in time recovery offers up to is being able to really let an engineer debug properly and kind of go back to a specific pain point time and be able to figure out what was the current shape of the data at that time and what can I specifically look into.
When you think about having all these different branches.
I think it's important to note that you don't have to worry about orphan branch costs or having an endless amount of branches live at any given time because there is a time to live of 30 days.
There is scale to zero built in.
So from a cost perspective, that isn't really an issue.
And yeah, during the POC, I was able to sort of test out unlocking this idea of utilizing FinOps to get a sense of costs of things you might not normally be used to.
And going back to the Spotify's internal developer portal of Backstage, you're able to see certain costs associated with...
different services you might have living in the portal, and get to see that all in a unified way if you're going to dive into the Databricks platform and use something like Unity Catalog as well.
So are these product features or ways of working or a mixture?
Because GitOps is a way of using Git.
It's not a thing.
So where does this fall down?
That's a really good question.
I think so.
There's the technology landscape and the surface areas of what the platform supports.
So we've been talking about branching as the concept.
And as Cam mentioned, each branch, if you're not using it, it scales to zero.
So I could leave them and not do anything with them.
And then there's zero cost.
If I go back to it and decide I want to pick it up again, then yes, then that cost will start to accrue because everything is cloud-based.
But I can prune those as I need.
So that's a technology capability.
Another technology capability is the ability to govern my database in a way that's uniform and set up policy and permissions and masking at the top level.
Right.
And then inherit those policies as I go down into each branch.
So if I am a developer and the question, I don't know if you had come up yet, but if I am developing and I say, oh, I can break a branch against prod, what does that mean?
I should try to be seeing everything on prod.
Does that make sense?
Like not usually.
So what do you do there?
Well, you set up policy, right?
Certain elements, certain columns actually need to be masked.
And maybe it's fine for a service principal to see more when you're running CI tests.
So that could actually be a different permission level that is not available to you personally as a developer, but it is something that you could do in CI.
And in CI, as part of my process, I'm doing CI with another fresh cut of my database bridge.
So all these sort of technologies are here.
So what does that mean now?
How do I take that and say, the technology is here, and now what do I do with it?
This is the actual really interesting intersect, and this is sort of why I'm so passionate about this topic in general.
It is because I've had that experience for 30 years before getting into data and AI.
And I do know what those pain points look like.
And I've been through Agile, coming from Waterfall, TDD, even XP.
What's missing is the methodology adaptation to support this.
And, you know, talking to practitioners such as yourself and suggesting, hey, the technology is here.
We need a new methodology shift as well to go with that.
So it's a paired approach, right?
You can lead the horse to water, right?
But, you know, you need to be able to drink and then leverage it in a way that doesn't make you thirsty, you know, next time you leave the pool.
Yeah, people are always the hard part.
So there's a number of patterns and some of them we've already touched on, but there's a number of patterns that you're trying to enable or dissuade, depending on which one, that if you don't mind, just either one of you likes to go into a little bit.
So we already talked about this a little bit, but eradicating the mock burden.
What does that mean?
I mean, I live in mocks.
Yeah, I can touch on that.
So, you know, back to this POC, one of the things I was able to claim was that the result was 20 to 30 percent of integration test code is mock infrastructure.
And so that's technically 20 to 30 percent that you could reduce if you were to just use branching off production and running your tests without mocks in this idea.
This kind of goes back to that original question you asked about developing from fiction.
So it's really just an optimization and allowing for development time better spent in a lot of ways.
Yeah, and just to add to that too, it's like we're not ruining all mocks, but the ones that interact with your database, your Rust service on down, those are the ones, that's the opportunity right there.
Because you have to stub.
certain things out because you may not have access to live services in other parts of your application landscape.
That's going to happen, right?
And you do have to service those in different ways.
But what we're saying here is that one of those services that you had to pretend actually doesn't need to pretend anymore, right?
So that's the one area.
The other areas probably still have to be unless you have been granted you know, API test APIs, like there's a parallel spin up from all your SAS providers and they give you an alternative version, a sandbox that you can play against and some do that you can ping out and destroy it or whatever, because it's not the real thing.
That is a luxury, you know, sometimes.
But now we've actually have this luxury for the database.
Cool.
I would say the continuous delivery person in me says, please don't do that with your external services.
The latency is not what you want in your pipeline.
But that's it.
That's true.
That's true.
So here you have a private link so that it's direct, right?
So that the latency is immediate, like it's sub-second.
And then I think we've already covered this one, but just because the pattern too is environments at Git Speed, just the ability to Git Checkout-B, right?
I mean, is it that simple?
It is that simple.
And so this one then.
These words in this order scare me.
Destructive testing without asking permission.
What are you all saying there?
Well, it is.
It's like on your branch.
Not on main, not even on staging if you're having an interior sort of environment, but on your feature branch, the one you just forked.
It's yours.
Do what you want with it.
And destructive testing is encouraged because why not?
You're exploring new edge cases that you maybe not had considered before because you were not afforded that luxury.
You had to share that environment, a dev stage environment.
Or if you're running embedded, and if you're running embedded, sure, you can do full destructive tests.
But embedded isn't going to, it's a proxy.
It's again, that sort of, it's maybe a closer fiction, right?
It's still not nonfiction.
Yeah, and I'll just add too that you could talk about this in the same vein as just having a place for experimentation.
And that's obviously very important from a development team.
And when you even tie this back to DoraMetrics as well, you might have failure modes that are typically uncovered from production.
When you're able to have this destructive testing, which can be free and easy to use via branching, then you might be able to have change failure rates drop because then you can uncover things in a test environment that you typically wouldn't find out until it's live in production.
And then the fourth pattern, again, we've probably already covered this one, but we're talking about the same primitive serves the agents.
Tell me about the LakeBase app dev kit.
Where is that?
How does that play here?
Ah, okay.
So that is an interesting one.
I'll take this one, Cam.
It's pretty much fresh.
You're hearing it first here.
Initially, when Cam and I started working together, I had developed a source control management plugin that worked with Visual Studio Code.
And it actually marries the concept of your database branch with your Git branch and also orchestration.
So what happens when...
I do that branch, and now I want to do a PR, right?
And how I want now to do a fresh cut when I'm going to go through CICD.
All of those operations are embedded through get hooks, in a way, within the IDE plugin.
And we could put a link to that later.
But what I discovered is that, well, you know, I sort of focused on, here's the methodology and the way that you can use this.
It's opinionated, right?
but it follows development best practices.
What do we, how do we take that and externalize that so that agents or say someone that doesn't use an IDE, right, can do the same, obey that contract and those primitives without the IDE plugin.
So there's an agent, there's a kit, it's a LakeBase app dev kit, brand new.
it was born from the plugin first so that agents can follow along with this paradigm shift as well.
And in fact, the plugin now has been readapted after the extraction.
The plugin now leverages the same toolkit as a library.
We'll share the link on the app dev kit.
Lots of great stuff around orchestrating workflows for your source control management, but also release management.
And then one that I'm adding.
very soon would be to follow the TDD workflow pattern and have all of your, your, the code creation part of it, be part of the test driven contract that we've all sort of, you know, have experienced with.
And this really plays well because your quality, just like everything that Kent Beck and others have said, this is the quality really goes up when you were able to put those things together.
I have, a test project that actually uses the TDD approach, the plugin paired together with TDD, and it works very, very well.
I'm having a lot less problems with my agents in code refactorings because I don't know how many people have gone through and leveraged CloudCode or others.
It might do a pretty good job if you have a pretty good spec, a plan from which to operate from.
But as soon as you start doing iteration on that, it quickly runs into and creates this spaghetti mess, this code mess that is unmaintainable.
And if you ask it, if you were working with an agent, you say, I want to fix this thing.
Well, it doesn't really necessarily understand the solid principles or other things.
And it'll create new code, parallel code.
It's a mess, right?
So having a TDD style operation on top of that and obeying more of the architecture guidelines, when you prepare those things together, you have a much higher quality code product.
So you touched on this a little bit with permissions earlier, but think about governance here.
Because access to production data is a multi-forked thing, right?
So there's, of course, the I don't want to bring down production.
That's generally considered bad.
But there's also things like personally identifiable information, things like that that we don't want.
Whether or not our feature branch or whatever it is is actually using it, we simply can't expose it.
And so is this like locked down by default and you open it up?
Is this open and you lock it down?
What's governance look like here?
So let's sort of visit maybe the DBA role in days of past and what we had to do.
I was a DBA once for, you know, I was being part of one of the roles I had.
And it's a job for sure.
And, you know, if you're supporting multiple developers, it becomes an overtime job.
Think of it this way instead.
Like, I still have my reviews for schemas.
But now I could do that asynchronously when I'm a DBA can do PR review just like everyone else can and see what database changes are going to be recommended or suggested and have an opportunity to say, no, I don't want that.
That's actually going to go break against the overall design model I have in mind.
OK, fine.
That could be a rejection of the DBA.
So that's a role for the DBA.
But the other role is really setting up the permissions model for different roles.
within my now database environment, but instead of having multiple databases that I provision, I do that at the top level and then at the branches that sort of inherit those policies.
And if I am a developer and I have a certain set of permissions and I can have attribute-based permissions, right, which also includes things like masking by default for certain elements, certain table, you know, columns.
And when I'm looking at that data or operating with it, that's my policy.
And I can't see that data.
Right.
And if I do have to do something in a field where I don't have permissions to actually see, well, yes, I don't have permissions to see it, but a service principle could when I deploy to a CICD environment and then they can sort of run through the tests that you couldn't.
And that's how you sort of establish the new paradigm and really change the responsibilities into becoming more of an architect on everything rather than gatekeeper.
And that's a big shift for DBAs.
So then the next one.
As far as challenges that this claims to solve, and I'm sure your claims are 100% valid, but production support reproducibility.
So what?
Talk to me about that a little bit, please.
This is actually a really interesting topic for me.
I came up in the days where I developed...
solutions around CQRS and event sourcing and event logs, things like that.
So what happens?
This becomes more truer.
So I never would ever want to replay all my events in an event log from day zero just to get to the point in time where now I need to discover where the defect occurred.
Who wants to do that?
Right?
Probably no one.
But what I can do, if I do have an event sourcing sort of model, and I am recording each and every transaction that occurs from every single consumer or whatnot, well, I can actually isolate that and say, okay, well, let me go to the point in time just before this event happened, branch the database, and then start applying the event log at that point in time, and then watch what happens.
So think about now my troubleshooting time to figure out how to solve for that defect.
I've just eliminated...
all of that burden of trying to spin up a fresh environment and trying to replay everything from from epoch into now a very quick turnaround of just the snap i needed apply some event log uh transactions get to that where and i can actually record that event and whatever i had transplanted and actually put that into my into my test bed so it never never happens again yeah and honestly to work off what kevin is saying it's realistically it's just a time save you are not having to reproduce an error you might have seen in prod and try and replicate something that's like the bug or that's something that's like kind of like the production environment you can actually branch in and replicate it exactly the same way so you're saving time and you're not having to play a guessing game essentially for support engineering yeah and you can automate that too right and that if i'm going to make sure I solve the defect, the problem.
I can then replay all of the transactions and events and make sure that my defect remediation had worked.
And I could do that all in a branch.
And just to go back to Dora again, this directly relates to mean time to recovery, time to restore to service when we're talking about the time save and how much more quickly a support engineer could diagnose.
understand a fix.
So, I mean, there's a lot more to this.
And I know you're, you're, you're all putting out a pretty detailed blog series of a few parts and some talks and so forth.
So I'm going to encourage your listeners to, to go to read more about it.
But I would like to ask each of you, you know, the age old question, what, what can our listeners do on Monday morning?
What's actionable?
So I'll give you each a turn.
You can go in whatever order you like.
But, you know, what's your piece of actionable advice that people can do?
It has to be a little bit more than just go buy Lake Base.
But short of that, you know, what can people do to learn how this might help them?
You know, in my case, I mentioned the POC I did with Backstage.
The primary reason around that was because I didn't really have a super applicable project at the time.
to test out like base on so my train of thought was well what projects have i worked on recently what have i liked working with what could be relevant and applicable here and that all pointed to backstage which i had liked working with before and so really like the thing to do on monday or the action item here is just to start experimenting and start recording your metrics and findings and like make a story about it and see if you like it and see if you think it's something that could make your life more efficient or save time or solve problems or constraints you're facing and it really just starts there with experimentation in a poc or you know if you do have an actual project you're working on for an account or an assignment with work See if there's a way for you to explore in that sense and maybe show off to your manager or your peers and show them this cool new thing you've tried out and all the learnings you have.
Yeah, I would echo that.
So, yeah, don't buy Lake Base, but try.
That would be my advice is POC it out.
Like go through and, you know, it's pretty cheap as it all things considered to try something out and, you know, look at like.
what we've been saying around measuring indoor attacks.
Like, what is it that you can do to say, well, what is my pain point in my environment?
What is it that I'm waiting on?
What gives me the most, what happens in sprint retro?
What are the things that really, like, I've never really addressed or actually just couldn't address and they've been silent?
Ask the question again.
Hey, if we had the opportunity to change something, what could we change?
And then what could we do to try it out?
Another thing you could do is just like, well, go back and look at like the number of mocks you currently have and look at that as opportunity.
All right.
To say, oh, my mocks continue to drift as I get further and further along and release and keeping those up to date are, boy, those are big pain points for me because now I have to really understand how my data model has shifted and keep those mocks in alignment.
Otherwise, I will test something that doesn't exist.
That's the fiction.
All right.
So count those up, see what the opportunity is.
And then I would say, if you have the ability to try something out, try it out.
There's an app dev kit that you can take and deploy locally.
You can operate on that headless without an IDE, or you can use the LakeBase plugin extension in your favorite, I'm saying favorite because there's really only two right now, VS Code or Cursor, right?
So others will be supported, I'm sure, but...
this plugin was more of a SAR as a POC itself.
So I encourage you to take that approach too.
Treat it as a POC.
See what opportunities it affords you.
Well, I'm excited to see where this goes because, you know, we were chatting about it.
We hit the 25th anniversary of the Agile Manifesto and other things that are happening.
And it was always like, yeah, we're going to do all this stuff iteratively except for the database.
Or we're going to do all this stuff iteratively except for the infrastructure.
You know, and so I'm really going to be watching closely to see where this goes.
And I want to thank you both for your time and encourage our listeners to check it out.
Thank you.