# Vanguard's AI-Driven Product Team Maturity Model

**Podcast:** Engineering Enablement by DX
**Published:** 2026-06-15

## Transcript

Welcome back to the Engineering Enablement Podcast.
I'm your host, Justin Riach.
This episode was recorded live at DX Annual, where we were joined by Kellyanne Pipe, Head of Developer Experience, and Nicole Scribner, Director of Engineering Enablement and Advancement from Vanguard, where they're working to transform how 800 product teams deliver value to over 50 million clients.
They introduce a maturity model they call Augmented, Accelerated, Autonomized, a framework for embedding AI not just in engineering.
but across the entire product development lifecycle, from product managers and designers to QA and operations.
They explain why treating AI transformation as an engineering-only initiative creates what they call the, quote, engineering bubble, where faster coding doesn't translate to faster delivery, and how Vanguard is setting an ambitious goal of five times faster cycle times by 2030.
Let's get into it.
Thanks so much, and thanks, everybody, for joining us today.
The conference has been going great so far.
And a big thanks to our DX friends for inviting Vanguard to the stage to share our journey.
So as was mentioned, we at Vanguard are putting together a maturity model for our teams.
And we're calling it augmented, accelerated, autonomized.
Later, I'll get into what that actually means.
First, I want to start with, in one word, how would you describe your org's AI adoption so far?
Think on that.
Hang on, we'll come back to this slide.
Keep those words in your head.
I'm Nicole Scribner.
I'm a director in our Chief Technology Office in Engineering Enablement and Advancement.
And I'm going to walk through the leadership lens of this journey today.
And I'm Kellyanne Pipe.
I'm the Head of Developer Experience at Vanguard.
I work directly with our engineering teams, and I'm really in the trenches, kind of building out the framework.
helping them implement it, running the pilots, and catching all the data that comes in.
For those who may not be familiar with Vanguard, we are a financial services company.
We are headquartered outside of Philadelphia, Pennsylvania.
Our mission statement is on the slide behind me, and we have 20,000 crew members who fully believe and embrace this mission.
To take a stand for all investors, to treat them fairly.
and to give them their best chance for investment success.
And we have 20,000 crew members, which we call our employees, rallying around this mission for 50 million plus clients.
So we asked before we started, in one word, how would you describe your org's AI adoption so far?
We asked some others before we came here, and we got kind of these responses.
A little bit all over the place.
The biggest thing was that it's accelerating.
We got it's awesome, it's overconfident, strong, risky, weirdly slow and fast.
Any of these sound familiar?
Here's the thing.
A year ago, our AI reality at Vanguard was this.
We had rolled out Copilot for engineers.
There were some pockets of brilliance.
We had some power users doing really incredible things.
We had our PMs not really using AI at all.
They were writing requirements the same way they always had.
And then we had our designers and our QA who were also not really touching AI tools for their products.
They were still doing their manual testing at the end of the sprints.
We had an AI engineering initiative, not an AI product team initiative.
And that's a massive difference.
A really important distinction, Kellyanne.
And although we love that our engineers report that they are 30% faster.
There is a big world to the left of the engineer.
We've heard a little bit about that world today in some of these other talks.
We have problem and solution discovery, and we have requirements in design that aren't moving at the pace of the engineers.
And to the right of the engineer, we still have some slow test processes.
And so we are not getting to production and getting that value to clients as quickly as we want.
So because we are focused and have been focused on the engineers, it's wonderful that they are more efficient.
But we're not seeing the overall improvement in cycle time that we had hoped.
This is what we're calling the engineering bubble, where the engineer is using their AI, they're finishing the things they need to finish in half the time, but they had been waiting for the PMs to give them the stories.
They're taking three days to a week to make the stories.
The designer has to handcraft every wireframe.
The engineer's knocking through that backlog real fast, and then there's no more stories coming up, at least no high-quality stories.
And then over here, after we've written the code, our QA has to go through it.
It has to go through all the test processes.
So this engineering bubble is when your organization equates AI transformation with, we bought everybody co-pilot licenses.
When your AI strategy is just an engineering tool rollout, it's not about adoption rates just for the developers.
We need to look at what's happening with all of the roles.
As a leader, and most leaders, I love data.
I love metrics.
I love seeing the metric that our engineers are 30% more efficient.
But the metric I love the most is in this black box, the output metric.
It's about delivering value to our clients as quickly as we can.
And that's what we're focused on.
So yes, as leaders, we ask questions about cycle time.
We ask questions about productivity.
But if we only have one input metric on the engineer trying to drive improvements in cycle time, we're not going to see that ROI that we hope for and our story falls apart.
So we've been asking this question, how do we help engineers code faster?
And what we realized is we need to ask a new question.
How do we enable our full stack cross-functional product teams to go faster?
And for Vanguard, that's 800 product teams across the enterprise.
And so now that we have this new question, We've also developed a new aspirational goal for these 800 product teams.
Five times faster by 2030.
What if all 800 teams could go five times faster?
And not by adding more headcount, not by outsourcing, but by really embedding AI across the whole product development lifecycle.
To do that, we need a map.
And that's the question that led to the map.
How are these 800 product teams going to know what to do?
What behavior should they emulate to get to five times faster?
Kellyanne, help us out.
So our map is an AI-driven product team maturity model.
That's a lot of words.
We have three levels, six dimensions, and it applies to the whole team, not just to engineering.
Now, this is a work in progress.
When we first started the presentation, it wasn't the three A's.
It was called something different, but it has always been the three dimensions.
We are running this past our IT leaders.
We're running this past our product teams.
We're running this past everybody involved in the process to really make sure that it's matching what we want to do.
And as our pilot teams get more familiar, we will refine, and we will continue to refine this model to meet the needs of our product teams.
So this is the three stages.
augmented, accelerated, and autonomized.
Augmented is where you're establishing those sustainable practices with some measurable productivity gains while you're managing risks appropriately.
So every role is beginning to use AI consistently for some things.
20 to 30% of the tests are getting some AI assistance, and we've got some shared standards that are being established.
AI is basically a helpful tool.
Accelerated is when AI moves from a tool to a real strategic differentiator.
We've got not just every role kind of using AI, but there's role-specific AI practices that are embedded.
We've got 30% to 50% faster delivery.
Autonomized is where...
it's really redefining what's possible.
60 to 80% of routine work is gonna be agent driven.
We have a two to three times capacity expansion with the same amount of head count.
So in addition to these maturity levels, we have six dimensions of maturity that we have outlined.
And I'm not gonna read the descriptions on the slide, but we are going to deep dive into each one of these today and share with you some of the behaviors that we hope our 800 product teams will adopt.
And we hypothesize that these behaviors are what teams need to improve.
So why don't we jump in to the first dimension?
Yep.
All right.
This is my favorite dimension, partly because it addresses everyone on the product team.
This is no longer just about the engineer.
So this first dimension is AI-powered delivery products.
So the engineers are still leveraging AI here to generate code, to generate their docs, to generate tests.
But we're also getting our product managers and designers and other roles into the mix here with tools.
Think of a world where the product manager is out in problem discovery, talking to stakeholders and clients, has their client notes, and AI helps those product managers translate those notes into workable requirements that then the engineers can take.
implement.
That is the world that we want to live in.
As teams get more advanced in their maturity, AI becomes the default.
We see teams experimenting more and we hope to see at least a 30% improvement from ideation to delivery to production for our clients.
When we get to an autonomized state, this is where it gets really exciting.
We still have our product managers working directly with the clients, ideating, understanding their needs.
But then we have agents in the mix.
And agents can then take that information and orchestrate from requirements, hopefully through to production.
We don't expect to hit an autonomized state this year, but it certainly is a North Star for us.
So you can have the best AI tools in the world, but if your code base isn't ready for agents, they're going to struggle.
So dimension two is around an AI ready code base.
At the augmented level, this looks like kind of the basics.
Every repo having a comprehensive README, an agents MD or a Claude MD file, or whatever the next iteration of that might be.
You've got linting enforced, you've got some unit test coverage above 70%.
CI CD is running on all the commits.
Your agents can navigate and understand the code base.
At Accelerated, you've got fast feedback loops.
CICD runs in under 10 minutes.
Code coverage is above 85%.
You've got comprehensive API docs, architecture decision records, security scanning integrated.
Agents can more quickly iterate through the code base because they get fast signals that their work is correct.
At Autonomized, they're generating complete deliverables from specs.
They self-test.
They autonomously refactor.
update dependencies, and improve code quality.
Your code bases are really agent ready at this autonomized stage.
And the key insight here is that the code base is really the interface between your team and the AI agents.
If that interface is bad, everything downstream suffers, no matter how smart the agent is.
Raise your hand if your code bases, if most of your code bases, most of your repos, have AgentMD or ClaudeMD in most of the repos.
I see some hands raised.
Now raise your hand if throughout your whole code base, you don't even have a readme in every single repo.
I see a couple of different hands raised there.
Everybody's kind of at a different stage of how ready their code bases are.
And across 800 product teams, we have way more than 800 repos.
We've got a huge code base.
So we have quite a way this to go.
even to really hit the first level of this maturity level.
And Kellyanne, that's definitely been a theme today in getting our documentation ready for AI and agents.
I think we've heard that through every talk.
The third dimension is around how much real work are agents actually doing?
This is making sure the agents are not just auto-completing, but they're actually implementing things.
The key shift here is that we're shifting from agents are assisting me to...
I am now an orchestrator of agents.
That doesn't mean the human role disappears.
The human role elevates.
We are in the early days here.
We're leveraging some agents that are provided by our SaaS partners and learning as we go to see how we can get these agents in an orchestration across the whole PDLC.
Now, this dimension is about what happens after you ship.
Dimension four is AI augmented operation.
Can AI help you monitor, troubleshoot, and heal production systems?
So in augmented, that's the monitoring and troubleshooting part of it.
You've got some basic monitoring and learning.
AI is helping you generate those postmortems when things inevitably go wrong.
In accelerated, maybe there's much fewer disruptions and much fewer need to develop those postmortems because you've got some predictive monitoring.
You've got automated remediation for 30% of your incidents.
By the time you get hit and autonomized, your systems are self-healing and 70% of incidents are auto-remediated.
You have near zero unplanned downtime and continuous optimization.
Kind of sounds like a dream.
Your teams can stop firefighting and start innovating.
All right, dimension number five, team autonomy and enablement.
And we won't, again, go through all of these bullets, but we definitely want to double-click on the behavior related to dependencies.
Kellyanne's been talking a little bit about agents.
We know agents are fast.
They're super fast.
They're faster than humans, right?
And you know what's not fast?
When we have to wait a month for an approval to use an API from another team.
Or we have to wait two weeks to get approval from security to this new application.
Or we have to wait to get an environment stand up.
When agents enter the picture into that ecosystem, it stops them in their tracks.
And what may have been tolerable for the human, now again, it just stops us.
And so we are really focused at Vanguard in trying to mitigate dependencies through a variety of levers.
We want to analyze where we can across our 800 product teams, understand what commonalities might...
exist and what's slowing people down.
Is it security?
Is it a vendor, et cetera?
And we want to tackle those problems now together as an enterprise.
And we've created an internal tool called our wait time analyzer that's analyzing all of this workflow data and not only giving enterprise insights, it's giving insights into our subdivisions who may have local issues that we don't experience at the enterprise level.
and that we don't necessarily address in our chief technology office.
So dependencies are so critical to get them under control and to mitigate them.
So again, dependencies that are tolerable at human speed become the bottleneck at agent speed.
And when agents can implement a feature in hours, every human speed dependency becomes very painfully visible.
And that's when we started saying dependencies are really defects.
All right, our last dimension, last but not least, responsible AI.
And this may seem counterintuitive because I think when we hear about responsible AI, people may think about governance and privacy and equate that to slow, a break, something that's in product teams' way and doesn't allow them to move forward quickly.
And sometimes we hear that.
We'd be a lot faster if compliance would just get out of the way.
But we've actually started to find the opposite to be true.
At this augmented state, at this early level of maturity, we're really focused on our crew having all the training that they need around our AI policies, around guardrails, around the audibility expectations, so they have what they need to build.
They can build.
without fear, they can work within those guardrails.
At an accelerated state, automated controls are doing a lot of the heavy lifting.
We're thinking about things like data classification for sensitive data.
We don't want that sensitive data to ever reach our AI tools.
And AI-generated code is auto-scanned.
We're testing for bias and fairness.
In an augmented state, which again is our...
our greatest maturity level here.
We again don't anticipate to hit this this year, but this is where we actually can leverage AI to help us with our endeavors.
And some of the guardrails is one, the guardrails around responsible AI, it's one of the first things we put into place.
We actually have an AI SDK at Vanguard that was one of the first things we brought in so that our teams have a safe place first to experiment with AI and now to begin building things off of AI in a place where they know that we're going to make sure that the vanguard stays safe.
And getting to that state of continuous monitoring and self-healing is definitely a place that we want to go.
All right, so we talked about these six dimensions and some of the behaviors.
It's a great framework.
It is a really great framework and we feel really great about it.
We also think we know the tools that teams are going to adopt to help them achieve adoption of these behaviors.
But like many others that we've heard today, I think Jennifer mentioned it this morning, and our friends from Mercari talked about this as well in their talk.
It's not the tooling that we're worried about.
It's the behavior change.
And we think that is truly one of our biggest barriers to adoption.
Changing hearts and minds is often much more difficult than bringing in new tools.
And so as a leader, I don't know if you can all relate with this.
I hear what you see here on the slide.
Who's saying, well, you're telling me that we are going to use AI to automate routine tasks.
What does that mean about my job?
Do I still have one?
Why should I adopt and skill up if it's going to take my job?
And what they're not hearing is, hey, we want your role to evolve.
We want you to focus on ideation.
and strategic thinking.
And really, the fear that creeps in is really is what preventing adoption in some cases.
And we as leaders need to help our crew and our employees understand, through example, that their roles are not eliminating.
They're going to evolve, just like we have over the years when technology has changed.
And the more examples that we can show our crew of where this is happening successfully across our 800 product teams, We feel that that will help us start to slowly overcome this fear.
The second problem that we have is the measurement problem.
The easy metrics are misleading.
And we have to remind our crew that we're not actually interested in those easy metrics.
Remind the leaders that we're not actually interested in those easy metrics.
We are not interested in the number of lines of code that are generated by AI any more than we're interested in the number of lines an engineer put into a PR.
That's right.
We'd love to say how much time was saved per developer, but trying to isolate that is very difficult.
We have to make sure that any measurement we do of the success of this is multilayered.
We can track adoption by are people actually using the tools, and then we need to also track is the use of those tools actually helping processes change.
Then we really need to track is those processes changing, driving the cycle time down?
Is it driving higher quality?
Is it driving more value?
The point of doing all of this, especially at Vanguard, is to deliver more value to our clients, to help more people reach their financial goals, and to be responsible stewards of the Vanguard.
And we have to make sure that any investment we put into AI really comes out with that outcome.
Shorter cycle time with higher quality and more value.
Okay, another audience poll.
What is your number one blocker to scaling adoption?
And two different questions here, two options.
Number one, is it people?
So things like leadership adoption and buy-in, skills, behavior change, culture.
How many people feel people might be their number one blocker?
Okay, a few hands.
All right, how about structure being your number one blocker?
This is things like tooling.
This is things like measurement, dependencies.
I love seeing this.
Some people don't even have their hand raised for either.
So I'm going to be finding you all and seeing why you don't have any blockers and what lessons you might have.
We get to append our slides with that.
So when we ask this question internally, depending on who's in the audience, the answer will vary.
If we have a group of product managers, typically we hear them say skills is their blocker.
If we have engineering.
they're talking to us about new and different tooling.
And when we have the leadership team in the room, there's always a lot of debate about measurement, whether we have the right measures and whether we're set up for success to measure.
Now, the great part about the framework and the dimensions is now that we have a common vocabulary to talk with each other about the blockers, and we can pinpoint these behaviors to really start to dig into what is holding us back in a more specific way.
We're going to quickly touch on some of the lessons we've learned as we've begun to roll this maturity metric out, this maturity model out at Vanguard.
And then there should be some time for some Q&A.
So lesson one is we need to meet our personas where they work.
Listening to the last talk, and I loved hearing that some of their PMs were hopping into Cloud Code and doing great things at the CLI.
We've seen that with some of our PMs.
Some of our PMs are like just pulling down terminal, popping up terminal, pulling down cloud code, making their PRDs.
Love that for them.
All 800 of our product managers are not successfully doing that.
All 800 of our designers are not successfully popping open terminal and feeling real comfortable going right into cloud code.
We need to really meet our personas where they are.
Oops.
Not every tool is right for every person.
All right, second lesson, and again, I think this is a lesson we've heard quite a bit today at this conference.
Tooling is the easy part.
Changing hearts and minds is the more difficult thing.
It takes time to grow at scale, and we have to keep with the same lessons and the transparency around behavior change.
One thing we recently did at Vanguard, which was a huge success, is we had one day where we brought our engineers together.
for a training or a conference that we called Prompt and Tell.
And this was a subset, maybe about 10 or so teams out of the 800 that showcased what they're doing with AI and specific tools, how they're making the testing process better, how they are building better designs.
And we want to do more of that.
The more all of our roles on our 800 product teams can see the success stories and see people leaning in and blurring lines for existing roles, the more successful that we'll be in our journey.
Lesson three, our agent speed is exposing a lot of organizational debt.
So here's some sample cycle time.
Our agents are driving implementations in two days, but their design review is taking four days.
And then that API that you needed to onboard to isn't giving you the onboarding for three days.
And then the security review queue is all backed up and you can't get your...
Stuff reviewed for five days.
And then finally, deployment of approval is going to take two days.
This is not actually how it happens right now at Vanguard, but it's just an example of how the processes that are in place aren't matching the speed of AI implementation.
Agent is fast.
The organization is slow.
So the way that we've always worked is now the thing that's holding us back.
And we need to think of how can we utilize AI to improve these processes where we're still having the same high quality standard faster.
And the last lesson for today, embrace responsible AI.
It is an accelerator, not an impediment.
It is not a brake.
Feels counterintuitive.
Again, we have to think differently about responsible AI and governance and privacy.
It's there to protect us and to protect our clients.
And we have to continue to share that.
The idea we can bypass this to move faster is really not the right sentiment because we're finding the opposite.
If you invest in responsible AI early, like policies, scanning, et cetera, you are able to move faster.
You're able to move safer.
And you can feel good about the quality of what you're delivering into our production environment.
Let's talk about what's coming up ahead.
What's ahead?
Okay, so a few key things.
I'm sure there's...
More than these, but these are the things that are top of mind.
The delivery lead or the delivery manager role is shifting and is changing.
Today, a delivery team, a delivery lead manages a team of people.
Tomorrow, we will have agents in the mix.
And our delivery leads and managers will need to think about that interplay between the humans and the agents from everything from workforce planning.
to resource allocation, quality oversight, workflow.
What does it mean to have both working together?
What can we expect?
How to plan?
We've heard it's really hard today to plan a year in advance.
So we really have to think about what that means and determine how we can train our leaders to be thinking about this one.
Second is that the measurement's going to catch up.
It's not there yet.
There's no perfect A.
AI ROI number.
If you know one, please come and tell me.
But there are leading indicators for sure.
There's, for us, as people progress through this maturity model, as our cycle time trends, hopefully, well, down for cycle time, up for quality, and as our quality metrics go up, together that can tell a compelling story.
So not quite one AI measurement, but together the numbers will begin to tell the story.
All right, and last but not least, 5x is ambitious, and we want to be setting a high bar.
Will all 800 product teams get there?
Probably not.
Will they improve their cycle time?
We bet that they will to some extent.
But the teams that are furthest along are already seeing, that have got some of these behaviors, are already seeing two to three times faster cycle times.
And the gap between AI mature and AI immature continues to grow.
We can't wait.
We have to act and we have to set that bar high now.
I just want you to think is what's one thing that you might do differently after this talk or after any of the talks you've heard today?
Whatever insights that you're gaining today is going to be the most valuable thing for you because it's your team.
It's your company.
It's the way that you work.
You know, our maturity model is probably not going to work for every company at this conference, but you can probably glean some insights and maybe build your own maturity model that will work.
just for your company.
And if we reflect on some of the other talks, there definitely are similarities that I think we can take away and reflect on and figure out how we can embed them into our processes back at our companies.
There's one thing that you take away today.
I think it's this on this slide.
This AI transformation is not just about engineering.
Be treated as an engineering initiative probably won't be successful.
It's a product team initiative.
We need to get all roles involved.
We need to get product managers, designers, and continue to have our engineers get more efficient as well.
Because when we optimize all the links in the chain, that's when we get faster.
That's when our cycle time will decrease.
And that's when we will deliver to our clients faster in a production environment.
That's it.
Thank you so much.
We have some time for Q&A.
You're perfect.
Thank you.
That was a really interesting framework.
I like a lot of how that's going to be able to guide different types of initiatives.
We have some really good questions from the audience I want to get into.
So you touched on this a little bit, but a lot of what you call out in sort of AI readiness seems independent of actually like using AI tools.
So did you increase investments like in platform and in platform engineering?
And how are you prioritizing the like boring work of getting the infrastructure ready?
I think what we're seeing with AI is some of the things that maybe we haven't addressed, things like certain types of technical debt, are definitely exacerbated in this type of world.
And we need to go back to some of our best practices around testing and requirements and make sure that they are visible for AI to absorb so it gets the full context.
We have not specifically hired more people or consultants to help us solve these problems.
We're looking to work within the product teams.
And we are trying to maintain a state where 80% of teams' work is around feature work that directly benefits our clients.
And about 20% or maybe 15%, 20% is around non-feature work.
And it's cleaning up some of those ills of the past that we are now facing.
We're also hoping that AI can help us accelerate and change some of those ills more quickly in this new environment.
We're actually encouraging, if you are coming to the engineering teams as a major tech tech effort, you know, everybody needs to change X thing in their code base for this new role.
Come to the teams with a solution possibly generated by AI or tell teams how they can utilize Claude to help them, you know, change their node.
Their node version, all that kind of boring stuff, it can really be, some of it can be automated.
That's really interesting.
It's kind of a meta perspective, I guess.
Cool.
All right, so on dimension number six, can you dive in a little bit more about the audit trails, like specifically what's included in those audit trails?
Yeah, this responsible AI is dimension number six.
And that is the one we are really just getting started with at Vanguard.
And the emphasis right now is on the training and giving everybody the foundational tools to work within AI.
I think in a more mature state, we're working towards a world where we have AI to help us continually scan for defects, to understand the health of our production environments.
I don't know that we have, you know, I think the tool stack there is open for what we use, but that's the idea is how can AI help our crew maintain the health of a product or an application, help us reduce the BAU.
Kellyanne, would you add anything to that?
Yeah, we, as we begin to roll out any of these tools, any of these best practices, we sit in the CTO.
We're also working really closely with our security partners as we begin to roll things out to ensure that, you know, what audits, what best practices should we be putting in place as these new tools come out, as these new practices come out, so that we are making sure everything stays secure.
And if we have these things in place today, what would the future look like?
And how can we...
Not just be as secure as we are today, but more secure in the future.
Yeah, and being part of financial services, our security bar at Vanguard is very high.
Very, very high.
And so we take great care in making sure that we have all the proper checks and balances in place.
There's restrictions right now in what you can use AI for.
We don't allow production data to go through AI until we are absolutely sure that the critical guardrails are in place.
to enable that kind of stuff.
Right.
A lot to think about there, actually.
How far along are your teams on this maturity map right now?
So I think there's three stages and six things.
Six dimensions.
Thank you.
Six dimensions.
I think it depends on the team.
We do have some pilot teams that hopped on the train real fast and intentionally, you know, we got a senior developer, a real...
innovative product manager.
And we put a tiger team on this and had them like run real fast and see what they could go with.
So they were able to come up with like a lot of success.
A couple of teams adopted some of their things that they had done and are seeing some similar success.
We have some teams where, hey, the tech lead really loves AI and is really clever and innovative and just kind of cracked the code in the way that that team operates.
So we have pockets of brilliance.
I think overall, most teams are across the six are kind of in that beginning phase still, maybe moving towards the middle.
We'd like to see a lot of the teams get to that first phase this year and see some frontier teams be out in kind of the middle phase.
We brought in Claude, I think August, September of last year, and that's where some of these learnings came from.
We were trying to have product managers go in and work in a way that engineers did, and that didn't always work.
And so I think we really truly started thinking about the different personas probably in the fourth quarter of last year and testing and learning with some frontier teams.
And I think we feel that we have enough learnings now to go out and test with a broader set of product teams.
That's good.
And it sounds like there's even an organic or even kind of grassroots component to find the heroes and replicate what they're doing.
Yes, that's right.
That's right.
How does Vanguard ensure cycle time measurement is consistent across 800 different product teams?
Do you have any guardrails in place specifically to prevent Good Hearts, gamification, that kind of thing?
We do have an ongoing conversation with our product teams to try and avoid gamification.
I don't think there's a perfect way to do it.
You know, you give somebody a metric, they're going to be like, I'm going to gamify that until I hit the metric.
We do try to avoid that by trying to avoid over-anchoring to any single thing.
So if your cycle time is really low, you're delivering things really fast, but all the things you deliver break constantly.
That's not successful.
If you're delivering really fast, but...
none of it's getting to the client because it's not what the client needed and your CSAT scores are still down, then that's also not successful.
So trying to avoid people just running at one of the metrics by making sure the other metrics are paid attention to too.
I'm just going to add on to that, Kellyanne.
We do have a great deal of senior leadership support on cycle time.
It is an IT-wide objective to reduce cycle time this year.
Every one of our subdivisions is at a slightly different place in their journey.
And, you know, these are 800 global teams and some have, you know, more aggressive goals.
But having that kind of IT lens and our CIO saying this is important really helps the 800 product teams rally.
And it's not perfect.
And I think our data is definitely directional.
And, you know, there's certainly outliers and we're constantly.
looking at the hygiene of the data to see if we can get better and more accurate.
And we also aspire to try to better measure end-to-end the product development lifecycle.
We've done a lot of great measures, measuring SDLC and the engineer roles, and we know how to do that.
But when we start with problem discovery, we haven't quite figured that out yet.
And so we're looking to expand how we're thinking about cycle time to start at the beginning of the process.
Got you.
Yeah, so that's sort of like concept to cash or ID to value, like I'm here several times.
That's right.
I wish we had time for more questions.
There's some really good questions in here.
But as always, I would encourage you to go find Kellyanne and Nicole in the audience.
Thank you so much.
Thank you.
Thanks for having us.
Thank you so much.
Thank you so much.
Thank you.