# AI Code Governance: Balancing Velocity and Quality

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-10-05

## Transcript

Five lines of code change, we can argue about for hours, 500 lines, you know, it usually looks good to me.
You only live once, so let's just get this change in.
When I hear people say, oh, there should be a human in the loop, I'm like, okay, but if you're not helping the human in the loop, you're really doing them a disservice and you're probably being a bit naive.
An engineer producing thousands of pull requests can look extraordinary on a dashboard.
Another producing a handful can look expensive.
Deciding which one is doing valuable work takes more than just PRs merged.
Joining me today to talk about it is Andrea Malagodi, CTO at Sonar, which provides AI code verification and governance for shipping code with confidence.
Andrea, how are you feeling about software engineering today?
And it's great to see you.
Thank you very much for having me.
Happy to be here.
What do I feel about software engineering today?
I think lately...
All the focus is on numbers and statistics, token maxing, et cetera.
And maybe somewhere along the line, we've maybe forgotten that software engineering is also a craft, right?
It's something that actually takes time to learn.
It takes experience.
Some of us have had to go through a few incidents over the years to actually get some benefits of learning, et cetera.
And AI is a set of tools that are incredible, amazing, can do wonderful things.
But it also takes a craft to learn how to use them and apply them in a way that gets great outcomes.
And I think right now, industry in general is trying to find out, like, how do I actually attribute the cost and the investment I'm making to outcomes?
I don't think anybody has figured out the, you know, the silver bullet answer here.
But I think all of us are trying to look at different aspects of the data that we get and that we can see.
to try to make sense out of, you know, the information we're getting and try to figure out where we should invest and where we should potentially pull back or modify a little bit the approach.
I think that's a really great point because we all get a little number obsessed sometimes and there is so much change happening right now within AI and within software engineering in general.
And I would be remiss if I didn't give a quick thank you to our sponsors this episode for helping us navigate that change.
Wallace Memory, Ingest, our season presenting sponsor of Chain of Thought, G2i.
But as we continue to explore the craft of software engineering, as we think about the numbers behind it, as well as we explore how AI tools can vastly impact the space and have frankly changed it massively over the last year or two.
I'm wondering, Andrea, from your perspective, how do you think engineers should be learning about the craft today?
You mentioned that in your opening statement here.
that there is a lot of learning to be done.
We can't just rely on these tools.
We can't just look at the numbers.
What should engineers be thinking about as they try to learn the space?
Yeah, I mean, I think, I guess there's two dimensions of this, which is the one on the individual engineer, where I think we've gone from, you know, somewhat structured prompting to using skills, lots of other instruments to try to instrument the AI to come out with the best outcomes.
you know, at the rapid pace that we can and obviously at a reasonable price.
So engineers need to kind of move up that maturity curve.
And I think a lot of people end up being a little bit stuck in, call it, you know, advanced prompting and, you know, have maybe not really jumped into the space where they're saying, well, actually, I'm handing work over and I'm expecting.
an AI set up to basically be able to solve that ask that I have from A to Z to get to a PR that can be shipped and basically go out the door.
And on the manager side or the leadership side, you know, we obviously get asked by people in finance to say, well, those numbers look awfully big.
What's actually going on?
Do you know what's happening here?
So a lot of us obviously spend time trying to figure out to crunch those numbers and see, well, what actually is going on?
And, you know, industry, I think, is going towards a cost per PR, which has kind of become a common number to look at to say, OK, what was the actual price for this PR?
I had N amount of sessions and it resulted in a PR.
And if I had this amount of cost and this amount of PRs.
you know, if I do some equalization on that, I get to a reasonable understanding of the cost of PR.
And we did that.
And what I observed in the numbers, which I had to say, I mean, it made me incredibly curious immediately, was I had on the one side, individual at the one extreme that had an enormously high PR count over 500, you know, in a very short time period.
And obviously, the cost per PR was quite low.
And I had on the other side, a person that had a very, very high spend, but very little PR.
And I was like, well, what's going on here?
And if I looked at it maybe from a pure FinOps perspective, I would say, well, black and white, many PRs, low cost, good, less PRs, high cost, bad.
Okay, those people that are bad, we should not have.
But the reality is when I then unpacked that, I found two engineers that were doing quite different work.
and had quite different problems at hand that they had to solve.
The individual that was doing this very, very high PR count had actually created this specialization, I would say, of having, call it a factory for themselves, of agents and a highly opinionated flow about what should happen in each of the stages of the work that these AI agents were asked to do.
The individual has basically said, listen, I'm going to come with a level of specification.
And before you do anything else, I want you to break this out and go into deeper detail of what all the different things are that are required to achieve a definition of done.
This then gets handed over in a second step of the workflow of actually doing some design and some architecture of that.
And this gets reviewed by that individual to make sure they're comfortable with this.
documentation records are written to make sure that that's persisted in memory for long-term purposes.
Obviously, there's coding and there's verification of all that code.
And then eventually we get to a PR.
And then, you know, there's also functional testing that happens and all that.
But, you know, this individual had crafted a very, very, very skilled, highly advanced, in my opinion, flow to optimize.
for being able to go at an enormous amount of velocity to get through, you know, massive amount of code that got produced.
Interestingly, with a ratio of tests to actual functional code that, you know, was almost two thirds of it was basically test and validations as well of all the stuff that happened.
So, you know, really quite sophisticated, you know, a very happy chap and, you know, always happy to share his experience.
And on the other hand, I found an engineer that had some very, very complicated problems around C++ and some of the work we are doing there that really needed a lot of very deep thinking by the AI that was very challenging for the AI and had to go in long, long turns to go through and analyze this particular work.
This work wasn't directly attributable to, okay, do all this.
you know, great, I get six lines of code, off the code goes necessarily.
So I think there is a story behind those numbers.
And it's important that we, you know, if you're leading an organizational team, that you really get familiar with understanding what are the different contexts that people are working with within and what problems are they trying to solve.
So yes, it's good to have numbers and we should be very careful to not pass judgment too quickly.
without really understanding the context of the problems that are being worked on and the issues that's being trying to be solved or the opportunity that's being explored.
It was a great learning for me.
And the best part of that probably was to get this individual that was essentially making using AI part of his craft and then being able to obviously share that to other engineers so they can take benefit for it.
Within that, of course, Sonar.
focuses on this fundamental idea of trying to guide the AI, verify the outcome.
And, you know, most of us work on brownfield applications of some sort.
So there is a backlog of things that can be solved by, for example, remediation agent that can go and look at all past issues, or now also hunter agent that can go and look at security.
issues that may not necessarily be found in the upfront workflow or be part potentially of legacy code that needs to be inspected anew.
It's funny because I think we're having this debate a couple of years ago, broadly as a community, about velocity and lines of code and people over-optimizing for them and understanding the value cycle of software and the differences between different projects.
And we're kind of back to having the same debate.
again, after I think we had this great run of, oh, let's talk about developer experience and actually like what's the value being delivered.
And then I think we kind of got re obsessed with the velocity because we were able to enable so much of it through AI coding and because I can simply code faster than any human can over time.
And it has opened up the opportunity to throw so much more code at a problem to solve problems faster with code.
Yeah, we've we've.
And it's changed the conversation, but it almost is making us fall back to fundamentals again.
I'm like, OK, what's the quality delivery?
Yeah.
And, you know, in the past, it was lines of code and we measured that, you know, lots of lines of code.
Fantastic.
Not a lot of lines of code.
Not very good because, you know, we're paying a big salary to these people.
So they should produce many lines of code.
And usually, to be honest, the people that removed lines of code were.
probably the most important people you had at the time.
And then, you know, we all got agile and scrumified and whatever.
And then there were story points, you know, a lot of story points was better.
So everything inflated, you know, but, you know, it looked good on numbers boards, et cetera.
And then, you know, here we are with tokens and token maxing.
And it's like, oh, let's have a leaderboard of that.
Again, I don't, I mean, you can't blame people for wanting to try to make some type of quantification.
It's quite normal.
in society that we try to quantify these things.
But we think we should be careful not to overreact to those things and try to take them, listen, it's an input.
And in the context of software, it is a craft.
And the moment you forget that and just make it into some type of balance sheet exercise, I think you lose a lot of the understanding of how systems come together.
Most of us that bank are very comfortable knowing that banks put an enormous amount of focus on quality and verifying every single thing that they do at the origin of that code when it gets you know the first line gets written out all the way through to very complicated large regression test suites etc and of course security testing and validation and so you know there is a a very important place for that in most software Although I would say some software today is getting to be a little bit throwaway, right?
You know, everybody has got AI.
Everybody builds beautiful dashboards.
They may last a couple of months, but they're probably not the business critical ones.
And, you know, for those, yeah, you need to make sure you don't do dumb things like expose your database to the world so they can go and suck out all the data, which has happened in a number of cases.
But, you know, yeah, quality is an important aspect of all software.
But obviously there's some that are much, much more important than necessarily others.
What are some of the lessons you learned from these comparisons you're doing as you talk to these individual engineers, as you do these studies?
I think you talked about the software factory that this one engineer has built versus the multi-turn and kind of deeper thinking approach.
How are you taking those lessons from these, you know, premier engineers to the rest of your organization?
Yeah, I mean, I think the thing that I've always concerned myself with is how do we take something that is a good thing in one case and try to figure out to make it broadly available to everybody, right?
So, you know, the thing that this inspired a little bit on our side was to say, well, can we think of a way that we could get every single engineer bootstrapped in a way in their terminal with having this set up so they can essentially work?
in kind of this higher maturity level way, which I think has a better outcome for everybody and, you know, a good scale economics in terms of what we are burning, in terms of tokens and in relation to the outcomes we get.
Important for us, of course, internally, but obviously also for our customers, is that we integrate the steps and the things that can reduce costs.
of tokens.
So for example in our solution we have an aspect called Vortex which essentially gives the agent the ability to pull out context.
of the particular code base that's being worked on, be that coding guidelines or the graph of that code, to essentially reduce the burning through of rereading all of that code by the LLM, which is kind of a little bit of a wasteful effort.
So things like that are important for us, obviously, to...
put into our own workflow.
And obviously that's also what we are trying to help customers get.
We don't want the long loop of, hey, I'll write a line of code.
I think this is ready.
Let's go do a PR, wait for that whole outer loop cycle to complete.
So we want to give customers the ability to get.
and our own engineers the ability to get an answer very quickly is there a problem with this particular file or these files or the files in combination you know almost at the terminal level so that's a little bit what we focused a lot on this year is the recognition that can we make that inner loop more economical and faster in terms of turnaround so you don't have to get the long life cycle of the outer loop completed before you get feedback.
So that's obviously an obsession as well to make sure we can make sure that everybody works consistently in terms of that.
Season four of Chain of Thought is delivered by Sphix.
We spend a lot of time talking about what agents need in production.
And one of the least glamorous answers is events.
Your customers want agent workflows that react to things happening inside your system, which means your API needs webhooks that actually work.
Not just a post request and a prayer.
retries, ordering, idempotency, replay protection.
Sphix does that as a service, and they wrote standard webhooks.
The spec that Anthropic, OpenAI, and Google bailed against.
So if your API doesn't have reliable webhooks, that's turning into a lost deal.
Join Brex, Drada, Daytona, and many others on Sphix.
Get started at link.svix.com slash C-O-T, or go to the show notes to grab the link.
Qualified startups will get $12,000 in credits, $50,000 for YC companies.
I can't recommend Sphix enough.
I'm a huge fan of their open source project.
I've actually contributed a bit myself and they're so easy to integrate with.
I think you'll really enjoy it.
Check out Sphix at link.svix.com slash C-O-T.
Season four of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents.
Your agent has learned your code base and how you work.
Now you want to use that context in another tool.
Exporting a file gives you a snapshot, but what happens when the context changes?
Walrus memory is a portable memory layer that lets your agents store context for later use across tools.
Python and TypeScript SDKs plus native MCP support let you connect it to your agents no matter where they are.
You said who can read and write, so sharing context doesn't mean opening up your whole memory store.
If you are building across tools or model families, take a look at Walrus memory.
Learn more at walrus.xyz slash C-O-T.
That's walrus.xyz slash C-O-T.
And I've seen you in other circumstances break this down into multiple, I believe you called agent-centric development loops that you're using.
Can you talk a bit about how you're thinking philosophically about what this looks like as we move forward in software engineering and how you've seen this be redeveloped for the current era?
Yeah, I think we call it guide, verify, solve.
So essentially we start, you know, at the beginning in the terminal or, you know, we hand something over to a terminal that's running somewhere else where agents can essentially interact with the code base and get this context that I spoke about, the coding guidelines, the architecture of the application, the intended architecture, and essentially get bootstrapped with a bunch of parameters on top of the functional requirements that needs to be done.
And work can essentially continue through those agents.
They may be collaborating.
You may have multiple agents that are kind of challenging each other.
But the result is something that you feel is ready to get to a commit stage where we can do a verification of that outcome before it kind of makes its way into the sausage factory of getting software out the door and get it shipped.
After that, we then do both an algorithmic.
and an LM-based analysis of the PR and the PR in context of the overall code base.
And we use both of those complementary techniques to make sure we really cover holistically all the different techniques and methods you could use to ensure that you have a very, very high quality standard and security standard of that code.
Because once that PR gets accepted, it's in your code base.
you know, that's your intellectual property.
And you want to be careful with that to not pollute it and, you know, get it to break down and deteriorate over time.
After that, as I said before, most of us work on some type of brownfield application.
They're changing very rapidly.
And in that, of course, we have got legacy issues that we need to go kind of and deal with.
Normally, that has been...
making its way through backlogs that, oh, let's have a hardening sprint to go and fix it.
Today, we offer a remediation agent that can essentially plow through this and raise PRs that developers can then look at and then accept.
So we're kind of taking a lot of that burden away and the cost of ownership of that technical debt drops drastically.
Another aspect which today, and we've seen this recently in news, but is this exploration of the code base from a security perspective.
And specifically here, we got an amazing code security team.
Kudos to everybody and them too.
And they have spent quite a lot of time researching what is the best way to traverse through the code base to find all these potential vulnerabilities that could exist, which accumulate potentially over time and haven't been explored in this way that we can do today in the past.
So you may have applications that may not have undergone a lot of change, but have an accumulation of issues that go across in the application stack.
Input sanitation is a famous issue in lots and lots of applications and lots of hacks and breaches has happened because of lack of sanitation.
Well, guess what?
They're still there, out there today.
They still get added and created and introduced into code base today.
Having a hunter agent that can really traverse through all that code base and find them and actually highlight the risk profile to get those issues remediated.
Obviously, you can give them to a remediation agent to get them solved.
But really making sure that you have confidence that that code base that you are operating on the basis of, you're probably operating your business, really meets the highest threshold of security is a super important aspect.
And, you know, we call those, if you will, kind of background agents.
You know, there shouldn't necessarily be a human assessment, you know, I want to go and do this thing.
So it is a little bit on a schedule.
It's automated.
It happens in the background.
And I think the biggest challenge for you know, anybody is that if I create a PR, it's the most important thing in the world.
If somebody else creates a PR and puts it to say, hey, Andrea, could you please review it?
Sure, when I've got time, right?
And this is a little bit the challenge with these things is that we get a lot of changes coming out of the woodwork and we have this backlog of things that we have to go through.
Guitar is awesome, for example, from our perspective in terms of being able to go and traverse through these PRs.
and explain what the hell is this change actually about?
What should I focus on?
What should I be worried about?
What should I plug into?
And by the way, are there actually any obvious issues here that need to be fixed before I even do anything?
So please go and fix them.
So you need this constellation and combination of tools to help you be efficient, to help make sure that you maintain the high standards of the intellectual property you already have.
You have to recognize that what you already have may already have a series of technical debt or security issues.
And you want to make sure to get rid of those.
The world is not getting safer when it comes to software in terms of, you know, things that can happen out in the wild.
So we want to keep an eye on that.
And then we also need to think about developers' ability to actually pay attention to the most important things.
Otherwise, it's just a wall of noise.
And I think I've said this many times before, five lines of code change.
We can argue about for hours, 500 lines.
You know, it's usually looks good to me.
You only live once.
So let's just get this change in.
And, you know, if you don't recognize that and just say, when I hear people say, oh, there should be a human in the loop.
I'm like, OK, but if you're not helping the human in the loop, you're really being a little bit, you're doing them a disservice and you're probably being a bit naive.
um if you're not providing some real assistance to them a ton of great points in here a couple that i want to pull on in particular so one uh coordination i think it's really interesting how sonar is beginning to set out these various lanes with specialized agents across multiple loops too so interested to talk a bit more about coordination security obviously but first i want to point out something that you said which is about prioritization both for individual developers and within the code base.
And I actually think it's gotten harder for us to prioritize because we're all being encouraged to assign out work to agents right now.
And so it's very easy to have 20 different threads going with multiple different PRs and process and code review.
And I worry personally sometimes that I am.
spending too much time across a variety of tasks, achieving a lot of work, but not necessarily prioritizing the top work that is going to be most impactful for me.
And something I've had to personally check myself on.
Are you systematically enabling that within Sonar?
And by when I say enabling, I mean, are you helping ensure that high priority work is prioritized first?
How are you coordinating this through the system?
Yeah, I mean, firstly, I'd like to say that I think proven the multitasking is an illusion.
It doesn't actually work and it doesn't produce better outcomes.
So I think we don't need AI necessarily to teach us that lesson.
I think it's just a life lesson.
Multitasking, it does not produce better outcomes just from that perspective.
Privatization is super, super difficult.
No doubt.
And I don't think that's an AI problem necessarily.
I just think the volume of possibility to actually have these multiple tasks handling or happening produces a optionality in terms of taking on more.
Right.
That's the expectation, perhaps.
And the reality is that our ability to prioritize between these different things.
is not particularly great, I would say.
And I don't think that's an AI challenge.
It's always difficult.
I've got a backlog of things that we need to get through.
There's a customer priority for this product feature.
We have this bug that we need to get fixed and squashed.
It's always challenging to do that.
And there's a little bit back to the software craft thing, right?
There's a level of experience about figuring out what is the most important thing.
For us, I have to say, we give a lot of freedom.
to the individual engineers, obviously, to look at what is the most important things.
And then again, also, when you think about vulnerability management and stuff like that, there are certain standards that we just must implement and we must obey by.
Certain regulation recently, for example, in Europe, requires very, very active engagement on anything.
that smells of vulnerability in terms of timelines, reporting timelines.
So you have to make sure you educate the organization on that and you make sure your policies and your procedures actually support it.
But in terms of agent to agent prioritization, I can't say that we necessarily have like a great framework that says this agent.
I think you need to figure out what your agent's MD file says about what the order of things and where the challenges should happen.
So if you have for, you know, if you find that's a good approach to personify the fact that there is somebody that is representing the product manager role, if you will, and coming out with definitions of done, well, if that part of the agent loop is saying, no, this isn't up to scratch in terms of what I expect in terms of definition of done.
Obviously, from that perspective, inter-agent wise, you could establish a prioritization order in terms of who has.
power to accept or reject certain things that come through that agent loop.
And on coordination in particular, I think we've seen quite a bit both from developers and their setups, but also broadly in the media lately as we've explored some of the agent forums that have been leveraged by agents forums to coordinate from misaligned evals and other circumstances.
The huggy face hack obviously being the big example of this.
What we've seen a recent like German Wiki forum that was used in others.
How are you thinking about the tools provided to agents and agent swarms more broadly as far as how we coordinate work going forward?
as humans step back to a greater level of abstraction?
Yeah.
I mean, I think the first thing that comes to mind to me is you should be careful that you're clear about what you're asking for and what the guardrails are that you're actually setting up for that.
There's obviously network tricks and we've seen them not work that you need to think about if you're sandboxing or other things like that.
I think for most cases, though, I mean, these are a little bit extreme in media and they're a little bit extreme cases of testing the boundaries of things that's happening in the frontier labs.
You know, in most companies, I think the benefit of a swarm of agents is that you establish a clear scoped problem or opportunity that you want to get explored and give that to a set of agents or it can be an agent that can spawn multiple sub-agents.
to be able to explore that in depth.
But to do that, I do think you need to make sure you're clear on what the boundaries, of course, of that is and how is it you set up your instruction sets around what it is you're expecting of outcomes and when is it you have a human in the loop being pinged to be pulled in to essentially verify, right?
It's super easy to go and say, you know, always allow and good luck and let's see what happens.
But I mean, you also want to be a bit mindful about like, hey, there is probably a natural checkpoint in the outcome of this work.
You may not necessarily know what the penultimate outcome is, but you know that there is a definition phase and you want to be quite clear about going through that.
And you should torture yourself on the definitions to make sure you agree with the definitions of done and the boundaries that may be generated by agents.
So that instruction set carries into the next phase.
or whatever that workflow is that you are trying to execute.
I think the risk is that people are becoming too permissive.
And maybe they'll say, you know, it becomes a little bit easier to become lazy, right?
Hey, go and solve this thing, you know, and then kind of always allow and let's see what the hell happens.
You know, that's quite risky.
And again, this takes, you know, with great autonomy also comes great responsibility.
And I think everybody that's involved in industry.
has to take that responsibility seriously and not just deflect that off to, you know, some AI engine that's going to come out and do type of a miracle.
Scoping is super important, making sure that you're clear on those definitions and boundaries of what you expect.
And, you know, from time to time, we see some surprising responses back saying, oops, I actually completely ignored that instruction and I went and did something else.
So there you have to...
hopefully you've got good security teams that are taking care of your developer environments and making sure they're boxed well and securely in relation to that.
And I think part of the challenge of this coordination too is that different agents, you know, depending on the context they're provided, depending on simply the fact that they're non-deterministic or even that they may have a different model supporting their loop.
may disagree on an outcome, may provide you different advice.
And so understanding how to handle these coordination challenges where they do disagree is a human problem.
What advice would you give the engineers or what techniques would you advise them to use to handle this type of coordination when agents do disagree?
Firstly, I think it's important that you actually set this up so that there can be some challenge because otherwise you end up with the risk of just getting a lot of agreements.
So I think firstly, you should make sure that you set it up so that there can be a challenge situation.
The other thing I find, and again, back to the example that we started with, is that I noted one thing, which was all the tasks were like quite small, right?
It wasn't, you know, go and build this enormous, magnificent thing and take four days and come back with it.
The person has actually been very thoughtful about shaping this into consumable bits.
where the individual felt, listen, I can still manage the output that's coming here and actually have an ability to go and have some input on it.
I think the risk is a little bit if you, you know, create 400,000 lines of code in an agent sitting that's running for five days.
I mean, you have absolutely no idea what's going on in that code base.
Now, if your appetite, risk appetite on that is that that's fine.
Okay, that's up to you.
But it's not something I would recommend that you do.
I think you should try to figure out to get those things broken down so that you have an ability to actually still have oversight and overview of it.
And, you know, governance is a lot to do with having insight into the things are broken down into components that you can actually manage mentally and also oversee.
So I would not advocate for.
you know, let's just take 400,000 lines of code and, you know, off with YOLO.
I think that's quite high risk.
But most of what I've seen that has worked very well has been constructing an intentional flow between perhaps an orchestrator agent that is using different personas, if you will, engineer, product managers, quality, et cetera, and ensuring that there's already a workflow.
expressed that ensures that there's good challenge between those different agents, if you will, or personas.
Because that very much mimics what we do in a way in human life, right?
Traditionally, you know, we build a lot of software, put it into a system, we had QA, hammer through it, send all the defects back.
Now, all of that can, of course, be done more or less in a session, if you will, between agents.
But there's still good principles.
of having, you know, check the checker type of setups, specifically on your functional requirements and functional validation.
This episode is sponsored by G2i.
I've said before on Chain of Thought that most teams still treat evals like unit tests.
Write them once, check a box, and move on.
That doesn't hold up once an agent is making decisions.
G2i is about to close that gap.
For over a decade, they've added and placed engineers at other companies, from startups to FAANG.
Two years ago, they turned that same judgment inwards, building their own bench to review our own environments, evals, and training data that models are trained on.
These reviewers know the difference between code that runs and code that's actually good because they've shitted it themselves.
If your team needs that kind of work and doesn't have the engineers for it, check the link in the show notes to bring them in.
Season four of Chain of Thought is presented by Ingest.
Long-running agents need endurance.
They must endure waiting for users and weather tool failures.
They must take each new jump in model intelligence in stride.
Ingest makes long running agents durable, observable, and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
Grab the Ingest link in the description or visit ingest.link.
slash C-O-T dash pod to get started.
Build for the long run with Ingest.
It's interesting you bring this up because I feel like for my personal development that I'm doing, and I should say personal software development here, I've taken on really like I feel like a technical product manager role in a lot of ways where I am kind of guiding the architecture.
I'm talking to my team lead agent, which right now is typically Fable 5.1 or Astra.
And I'm saying, OK, like, here's what I'm envisioning from the product.
Here's what I've seen from user feedback.
Maybe I'm having them assist me doing research.
And they're really acting as my engineering team leader, engineering manager, and they're managing a fleet of sub agents.
And personally, I like to have multiple different model families involved in this, particularly at like review steps where I don't want it to be all open AI agents or all, you know, anthropic agents that are doing the work.
the work.
I want to have a few different model families.
I'm using like Inkling, MUSE Spark, Grok and a variety of other open models, Nebotron, etc.
to come in on the review step and say, OK, let's challenge the assumptions, because I do think there's this interesting phenomenon where just like how you can see a company's engineering team start to have biases within their work.
If I only throw GPT agents at a problem and then I have GPT agents also doing the review step, they often have similar biases or they support the same direction that these other agents have.
Whereas if I have, you know, I keep those same that same team of 5.6 soul led by an Astro team lead agents that go and build the project for me.
But then I have, you know, Fable 5.1 and Opus from the.
Anthropic family and I have like a Grok agent review or having you spark a review, they will provide very different feedback because of this cross model family review.
So I'm curious if you're seeing similar patterns in your coordination and how you're layering models within these loops, whether it's the verification, build loops, etc.
Yeah.
I admire your credit card company.
So yeah, good.
Yeah.
I think what I've seen work well, I do think we get a propensity to get a little bit stuck on single models, right?
Oh, you know, I kind of know what I'm going to get back.
So I kind of like prefer that way of working.
And it's an interesting thing when people change that.
Oh, OK, this is a quite a different experience.
I agree with you having, you know, obviously these higher class models to try to do that planning, do the architecture, do kind of the set out of specifically.
complex definitions of done because you want to get make sure you have the functional verification in mindset as well and then you can farm out to use different agent different model families to do you know different parts of that work and i like um also this idea of uh the challenging from different model families i don't know if i've seen the efficacy on you know is there really enormous value in the long run in terms of that?
Or are we kind of having this sense that because I'm asking Grok, I'm probably getting a different opinion.
How do you separate and segregate out these things?
So long-term and at enterprise scale, for example, you could do that.
Could be quite a challenging situation.
If you've got 10,000 engineers in the organization, setting up all these different models, managing zero data trust, et cetera, it becomes quite complicated.
I do, though, think that obviously the move towards local, hosted and stored models that you're running specifically also just for token costs, you know, it's not just a thing, it's happening and it's a reality so that people will be mixing perhaps, you know, frontier models with more, you know, with open models that they may be using and hosting internally and mix and match in that way.
And I think sometimes we need to remember, you know, if you're trying to manage 10,000 engineers or a situation with that type of setup.
it's some quite complicated problems you get into just in terms of scaling these things out.
Obviously, I would always recommend, hey, try not to have your LLM redo all the context work.
So you should use something that can give that to you.
We've got a great solution in terms of Sonar Vortex.
And also the verification.
I mean, you should focus on that functional verification.
and try to figure out all the different code quality security challenges solutions like sonar can help make that faster and cheaper for you um and yeah obviously the mix of a deterministic uh technology with non-deterministic to get a holistic outcome uh i think it's a good good option i think a good point from what you said uh is that myself and the folks who are truly on the frontier of this are often talking about individual or small team setups, or we're listening to people from the model labs where they are getting unlimited tokens to leverage.
And that is not always the case for when we're trying to implement this within an enterprise or within a small business even.
There are coordination chain challenges that come in here.
There are cost challenges that come in here.
And it's really easy for me to say, oh, I have an OpenCodeGo subscription, Claude and Codex.
And like, oh, that gives me access to so many models.
And I use OpenRouter for a few more free.
It's not that much coordination costs for me individually.
Like it's a little headache occasionally.
But they can pilot themselves in a lot of ways, at least for the review steps.
Whereas if you start to scale it out to a team of four people even, or a team, let alone a team of 10,000.
there are significant challenges that start to come into this.
And I know you have some data behind this from, you mentioned SonaVortex, you've done some studies around this, you've done some studies around the coordination tasks.
What are some of the key data points that you might want to bring up around coordination challenges and what you're seeing within AI loops today that maybe we haven't touched on so far?
The rollout of these things at scale, I mean, it's a super, super hard.
challenge, right?
And again, as you said, having a small team can be hard enough.
You know, even internally for us, using lots of different model providers, like how do you give to your finance team a single view of all that cost that's coming through the sausage factories, right?
You know, if you will, how do you just produce that?
I mean, in this particular case, you know, I personally had to write.
a bytecode, you know, some type of cost analyzer that pulled all this data in from all the different APIs that were available and try to quantify things so that finance could essentially figure out, OK, but actually, what is it we are spending our money on?
Secondly, if you just had a team of 10 engineers working on the same code base, how many times are we reading and rereading that same code and paying tokens for that?
And, you know, I mean, if you're sitting in one of these frontier labs and you've got, you know, tokens galore, it's all well and good, as you said.
But, you know, if you're a smaller engineering company, maybe you've got like 20 engineers or 30 engineers, and suddenly now you've got a per developer cost.
Let's just imagine it's $1,000 per engineer per month.
Well, I mean.
there has to be revenue on the other side that's going to cover those bills or you know bad things are going to happen so you want to be very sensitive to each of those tokens and what they actually get used to what we found in our studies in terms of context specifically because that's really where we see people can kind of burn uh quite a bit of tokens that's kind of i mean is it waste i don't know if we've called it waste necessarily but it's needless in a way, and you can do it through CPUs at cheaper rates, is that cost savings can be up to 30%, for example, just on reducing that token cost and that token spend.
So rolling this out at scale...
I mean, you have to be quite conscious of what it is you're putting in motion because once that train goes, it's very difficult to pull it back.
If you had unlimited tokens and you hadn't thought about necessarily setting up some budgets and figuring out to setting in configurations that allow people to have a lower token cost and you suddenly start saying to people, well, now you can't spend anymore.
They've already now developed habits where they're like, hey, hang on a second.
I can't do the things I was used to do.
And, you know, in the interviews even, I mean, I sometimes get asked, you know, what's your token budget approach?
Because they come from places that have had to do very strong restrictions because things just ran away.
Or alternatively, they're getting incredibly small budgets, like $50 a week, et cetera, which is like quite difficult to handle and deal with.
So at scale, there's a lot of consideration you have to make for this.
In my prior employers, which have like thousands and thousands of people, I mean, this is a full-time job for a large group of people to worry about and make sure to manage this, specifically because the correlation between a token burnt and the outcome vis-a-vis a revenue stream is quite difficult to actually connect.
And I don't think anybody's found the silver bullet.
So you can see that there's more productivity.
You can see your PR is going up.
But in the end, is that then converting into more customers, you know, more sales, whatever it is, business you're in.
Yeah, that's still a challenging thing, I think.
Yeah, to our earlier point about connecting the code to value delivery, we've been having this problem for years.
It is still a problem.
Yes.
Just the abstraction layer around it has changed a bit.
Yeah.
I would say the other thing just to add, which is a little pet topic of mine, just because I carry a couple of hats, is at the same time, we've also given a lot of non-engineers ability to create things.
They may not necessarily know there's code behind it, but to create things, which means that the whole problem of shadow IT, at the same time as this pressure of trying to figure out frameworks that you can roll out on an enterprise level, those same IT organizations are also challenged with, well, finance used to create these spreadsheet with macros, shadow IT, if you will.
Now they're creating 100 dashboards and giving them to executives.
All of them have to be secured.
Data has to be secured.
They have to have maintenance and servicing.
They may be using vulnerable libraries and, you know, the individual in finance may not know even what that is.
All of that also has to be managed and put into a framework on managing.
So those challenges are big for all enterprises to be able to manage that well and deal with it.
Another big challenge that we have touched upon in this conversation that is exacerbated by the amount of code we are now shipping is security.
I mean, we have seen the fact that it's much easier for LLMs to change security vulnerabilities.
So smaller vulnerabilities matter much more than they used to.
We've seen the fact that they can simply operate at speeds that are unprecedented as far as an attack occurring.
And now we're also simply shipping more codes.
There's more surface area to actually defend.
What approach is Sonar taking to this, I guess, new defense in depth there as I'm thinking about it?
Yeah, I mean, our product has always had, you know, assessed analyzer.
And actually, we did some quite innovative work of analyzing dependent libraries vulnerabilities that when you bring those libraries into your application, and if that library has a vulnerability and you're calling that particular function, we essentially understand through the analysis that we do that you have that dependency tree of issues.
And on top of that, we released recently our Hunter agent that essentially explores the entire code base for these vulnerabilities.
And in this case, using LLMs exactly as you just said, because the chaining and the ability to look at these issues in the compounded effect of them, you know, is something that LLMs can do incredibly well.
So this is something that we...
obviously it made into a product and you know we now offer to customers it went live uh last month so you know a lot of interest in in terms of that uh in terms of internally obviously we are um just as anxious as everybody else And, you know, there's a lot of perimeter controls and there's a multilayered onion of figuring out how you manage your vulnerabilities, etc.
I think the concern that everybody has is just the onslaught of issues and prioritization, as we spoke about before, is a challenge.
And figuring out to do good triage on these particular issues and vulnerabilities to make sure you are dealing with the most important ones first.
is obviously also something that we spend a fair amount of time on as a company that services lots of other companies.
Are there particular lessons that you've learned from the larger security incidents around agent X-Warm coordination that we've seen in the news?
What are you learning from this, I guess, the frontier of security vulnerabilities?
Yeah, I mean, like you said before, details matter.
right?
Small vulnerabilities and small issues matter.
We may not, historically, if we had, you know, a penetration test or a bug bounty, the humans that use lots of cool tools to do all of that may not have been able to conflate those things together to actually explore them.
You know, so I think the attention to detail and the attention to the small issues.
is super important so if you don't have an ability to penetrate through your code base and go and find those i mean you have to hurry up and get that put in place as fast as possible otherwise the compounding effect of those small things like i mentioned before input sanitation is a famous one It's not new.
It's been around for a pretty long time.
But there is a remarkable amount of those type of vulnerabilities, cross-site scripting.
And, of course, the LLMs know about all these type of vulnerability classes.
It's going to use that knowledge to say, well, okay, why don't I create a battery of opportunities to go and explore?
And because it can do that at an enormous rapid pace.
it can get through a tremendous amount of possibilities in terms of exploring code bases and their vulnerabilities.
And at the end of the day, they are just fundamentally using the techniques that we already know today.
It's just they can do it, as you said, in a swarm.
So instead of a human that's maybe doing two of these, they can do 100 at the same time or 1,000 at the same time and really explore it.
So if you haven't hardened your surfaces and you haven't hardened your fundamental code base, you are going to get issues.
It's definitely a new era.
No question about that.
We simply have a pace and scale that we've never seen before.
And it seems to be continuing to accelerate with bigger models, with more tokens being thrown around, with these software factories that folks are building.
And I think it creates a situation as we kind of go back to this coordination problem we've talked about throughout this conversation.
but also to the multitasking challenge that we brought up briefly earlier, where it's really easy to get distracted from the most important things your organization needs to work on because there's so much going on.
There's so much code flying around.
How do you preserve time for things like investigations when something does go wrong?
I mean, I think you I mean, this is not necessarily a new problem.
I think, you know, in the past, we used to say, OK, there's a team over here that take care of all this.
And then the engineers are working on this.
Bit by bit, you know, obviously everybody realized, well, it's kind of not a great model because then basically we just externalize the problem.
Best place to have it is with the engineers that actually produce the code because they own that code base and they own the service.
And in the same way here, you need to have somebody that is on that rotor of doing that gardening duty of going through and doing triage.
And it's a little bit the internal defender, whether you do that in.
you know, a model where you say there's a certain amount of days or weeks or whatever that person is doing that role.
I think that's up to teams to decide, but they should decide to have somebody that's taking care of that inbound noise traffic that can help shield the team and rotate that role.
So everybody gets expertise in seeing what the different patterns are of incoming noise and to be able to find the signals that are important.
So I think you do need to have that role so that the team can be partitioned into people that can do deep work and focus on that work.
And there's teams that can do the on-call, take all the noise and figure out to find the signals.
And of course, you need to use a bunch of tooling.
And the fortunate thing for us is, I don't know if you've heard, but there's this thing called AI, which is actually awesome at taking that and helping you in that.
And many teams develop a set of skills.
or flows that has to take that data in and helps them get to the essentials.
You know, we had a situation with hundreds of CVEs being reported, but, you know, we knew that there was a high likelihood on a number of those that it was false positive and, you know, it was noise in terms of that report that we got.
So, you know, the teams have built skills and capabilities to basically troll through that and make sure to explore to say, well, okay, which one actually matters?
And I think most teams develop those type of little utilities and helper tools that can assist them in also keeping control or keeping oversight of all the stuff that's incoming.
But it is definitely challenging, right?
And it's only going to become more.
Well, now apparently we're having a big AI slowdown, but let's see what happens over the next couple of months.
But yeah, it's definitely a challenge.
And I think teams...
you know, have to be creative and look at opportunities to also use AI to get them help.
And the best of teams have, you know, a set of agents that help them triage that information to go and get to the essentials.
And hopefully to a point where they can actually go and say, well, let's get the essential and let's get fix and get the fix shipped with as little interruption as possible.
And I think an interesting point from what you just said is that we are in an era of personalized software, both for teams and individuals, because code has gotten so much easier to ship as far as having an agent drive it for you.
Whether or not it's secure, we can we can we've had a whole conversation about that.
But there is an opportunity to personalize and provide apps that you wouldn't have otherwise been able to spend the time to build because you have too much going on to help accelerate your processes, understand what you're getting up to and do exactly what you want in your workflow.
And I think it's really exciting to have these conversations about how we build both the broader loops for enterprises and also individuals.
It's a it's a really magical time in engineering right now.
And personally, I'm finding it very exciting.
Yeah, no, I mean, in crisis, there's opportunity, they'd say, right?
And the same is true, you know, in software, it is a little bit of a crisis because of the volume of information opportunities and, you know, things that are coming towards us.
But that creates opportunity to be innovative and create new things that we may not have, you know, imagined we could do before.
And, you know, I...
admire and welcome all that creativity.
I see it constantly, engineers coming up with ideas and things.
I'm like, wow, you know, I hadn't even thought about the fact that we could do this.
And so, yeah, it's a really exciting time to be in software engineering.
And I think the good news story, at least for software engineers, is it doesn't look as if everybody's going to be out of a job.
I think, in fact, actually, companies are hiring more and more software engineers because of the opportunity that AI represents.
So, yeah, I'm happy to see that the craft is well and alive.
Andrea, thank you so much for a fantastic conversation.
Where can folks go to follow you and your work?
Follow me?
Well, if you go to sonarsource.com, that's the most important thing.
And, you know, I'm obviously on LinkedIn.
And then I got my personal photography website, which has got nothing to do with AI.
But that's up to people if they like that kind of stuff.
We will link on the show notes.
That sounds fantastic.
I will check it out.
Andrea, thank you so much for a fantastic conversation.
And thank you to our listeners.
If you enjoyed this conversation, don't forget to drop a comment.
Let us know what you thought.
Tell us what we missed or just tell Andrea, you know, he looks great on camera.
We'd love to hear from you.
It's always fantastic.
And Andrea, thanks again.
Thank you.