# Agent Skills: Enterprise Context Management

**Podcast:** The AI Native Dev - from Copilot today to AI Native Software Development tomorrow
**Published:** 2026-02-03

## Transcript

A lot of hype around it right now.
Skills.
First of all...
This isn't actually a new thing at all.
Cursor rules, I can't even remember when that came out.
Claude skills, again, something that's been around.
So skills are basically a standard unit of context to provide to agents that teaches them to do something, so to acquire a skill.
Sad reality that people are copying skills all over.
They're designed to be reusable, and yet we copy and duplicate them and copy everywhere.
So that's not awesome.
When we think about adding skills as a first class citizen into our organizational process, What do we need to consider?
You have to think about how do you own them long term.
You're better served by thinking of skills not as a markdown file, but as a unit of software.
You have to think about how do we maintain those skills, how do we keep them up to date, and so it's kind of a whole life cycle management of it.
Really exciting that TESL last week released support for skills.
There is coverage and functionality for...
evaluating skills, distributing skills through a package manager's style mechanism, and also support for skills in a more complete lifecycle way.
And all of those combined to just create this flywheel and now there's huge excitement over something that we've believed all along here at TESOL, which is context should be reusable and there's a difference between intelligence and knowledge.
Before we jump into this episode, I wanted to let you know that this podcast is for developers building with AI at the core.
So whether that's exploring the latest tools, the workflows, all the best practices, this podcast's for you.
A really quick ask, 90% of people who are listening to this haven't yet subscribed.
So if this content has helped you build smarter, hit that subscribe button and maybe a like.
All right, back to the episode.
Hello and welcome to another episode of the AI Native Dev Podcast.
My name is Simon Maple.
I'm your co-host for the episode and joining me is Guy Pujarni.
I'm also co-hosting here, CEO and founder of TESOL.
Wonderful.
And today's going to be a nice special episode where we're going to talk about something which is, there's a lot of hype around it right now.
Skills.
Agent skills, just general skills that we're going to be talking about.
What is a skill?
How should they be used?
When should they be used?
What is a skill compared to other types of context that an agent may or may not use?
And also, what are the certain things that we need to do to make sure a skill is...
right, a skill is good, a skill is accurate, and also some sneak peeks into some of the great things that Tesla announced last week on our support for agent skills and skills generally.
So there's a lot to pack into this episode, Guy.
Let's get started.
Yeah, lots to talk about, and skills are all the rage.
Absolutely.
So why don't we just define what is a skill, Guy?
I think that's sort of an interesting question coming from you.
It's like I sort of thought that you're sort of a skillful person.
No, I'm a highly skilled person, but I struggle to define it.
Lots of people find it hard to put their finger on it.
Indeed.
So maybe you learned something.
So skills are basically a standard unit of context to provide to agents that teaches them to do something, so to acquire a skill.
They have initially been introduced by Claude as part of Claude Skills.
capability, although they were very, very similar to things like cursor rules or other built-in guidance that existed in other agents before.
But still, Claude Skills paved the path to definitely this as a content.
And effectively, it's just a standard file structure.
that defines a skill.md file that has some definitions of what this does and some metadata about when should the agent use it, alongside with supporting files designed in a certain way so that the agent can acquire that skill, use it, and do so in an LLM-wise way.
For instance, a core element is this progressive disclosure of having bits of knowledge that the agent can traverse through so it's loading the right information at the right time and doesn't overload the context window.
Yeah, a couple of interesting things that you said there.
First of all, this isn't actually a new thing at all.
You mentioned cursor rules.
I can't even remember when that came out.
That was a while ago.
Claude skills, again, something that's been around for a little while.
Since September, I think.
Since September.
So it's interesting that it's kind of like, you know, heating up right now.
Another thing that kind of really interested me there is when you talk about the skills MD, it's just plain text again.
We're just talking about, you know, you're passing some text to an agent.
And LM reads that text and then does things based on that.
So there's a lot.
really in terms of us being skillful as well, about how we construct that, how we write that.
And I think we'll talk a little bit about evals.
So why have skills all of a sudden become an overnight success and everyone's talking about it and everyone's releasing various skill support?
What's happened?
Well, the key thing that happened is, similar maybe to MCP, is that it's become a standard.
So Anthropic, very late last year, have kind of introduced an open standard.
Standard is a bit of a...
Funny term, they've kind of coined one.
They called agent skills, and they published the definition of it.
And a little bit similar to what we've seen with MCP, the rest of the industry lined up actually fairly quickly.
And so now within a short span of time, maybe within a month, you've seen comments from Cursor and Codex and I think Gemini and others about supporting.
skills themselves.
And what that does is it suddenly gives the users, the creators, anybody that's actually kind of looking to use agents, gives them an ability to create one thing, the agent skill, and roll that out.
And so you've seen at the beginning of the year, you've seen Cursor make statements that they will embrace skills and actually phase out.
Again, they're very similar in many of their traits, so they will phase that out.
And you've seen this explosion of ecosystem.
The other very smart thing that Anthropik has done is that they created a create skill skill that just made it very easy for people to start accumulating it.
And maybe the last important thing is that skills actually were initially introduced in Claude less as a development thing and more as an ability to create some instruction for non-developers when you're using Claude to whatever, create an Excel sheet, right, or perform some type of musical review, right?
And so now that approach is true for the skills and agent skills as a whole.
And so it also introduced kind of a lot of non-developers to...
kind of this opportunity of creating kind of reusable artifacts that they can share across the team that has been very, very exciting or like reusable workflow.
So a little bit of a taste of software development style capabilities of reuse to the non-dev world.
And all of those combined to just create this flywheel.
And now there's huge excitement over something that we've believed all along here at TESOL, which is context should be reusable.
It should be something that you don't want to reinvent the wheel.
There's a difference between intelligence and knowledge.
Yeah, oh, absolutely, yeah.
Nice little takeaway there.
And I think, so let's go into a couple of things there.
First of all, I've actually been doing a little bit of work with some of the folks at CodeGuard, and it's an amount of context which is talking about how you can actually build, how you can review code in the most secure way following very specific security rules.
And I think one of the interesting things or one of the things that they saw as a big drawback was the fact that if they want to produce this, they know that the people who are consuming it, some are going to be using Windsurf, some are going to be using Cursor, some are going to be using Claw.
And so they actually need to structure it in multiple ways so that every different agent can consume it.
And I think this creating this standard that all agents can kind of like build from allows you to effectively be able to deliver one thing, fingers crossed, hopefully, as long as everyone starts supporting it, that people can then just kind of pull in.
So I think that's a really important piece.
Let's talk a little bit about context and context engineering and where skills sit in that, because really, skills are, you know, they're a part of overall context engineering.
What would you say are the differences between skills specifically and general context that an agent or an LLM can use?
Definitely there are many ways to basically introduce...
more words into the context window of the agent.
We need to remember that agents eventually still remain just interfaces to the LLMs.
So every time there's a request to the LLM, the question is, what is the data?
What are the tokens that get put into that?
And so all of these different means of context engineering, they're just about giving the right path to choosing the right words to include in the message.
So in that sense, they're all...
They're all a zero-sum game, right, of what you put in.
And you can easily put too many things into the context window that if you have so much context, so much set-the-groundwork that the actual instruction gets lost, or too few things so things don't get known.
So all of that is sort of the constant exercise.
When you're talking about external context that you provide, I'd say there are sort of three primary buckets.
There are rules, which you kind of...
shove it into the context window whether you like it or not right by hell or high water you would put it into that ClaudeMD or into a must rule a must use rule in cursor, although cursor sometimes ignores them.
And those are mandated.
They are very important things, but they take up context window space.
So you can't put full on documents this way.
It's expensive.
MARK MANDELSONIFFIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIENIERIEN to guide an agent, but you have to be limited in content.
Oftentimes, they are an initial instruction and some link, some pointer to another location.
Then you've got skills.
Skills include, like cursor rules, basically a tiny rule, like a little bit of data that goes into the context window for the agent to choose to invoke a skill implicitly.
You can choose to call a skill actively, like a command, in which case you don't need those things there.
You can also, you expect the agent to invoke the skill at the relevant time.
So you have to put a little bit of that breadcrumb in the window.
There's some development right now, but maybe separating that out within the agents.
Does it go and consult some directory?
For now, that's the reality.
And then you've got docs.
Docs are just available information for the agent to find, but they're not naturally findable.
for the agent.
And so you need to either leave a breadcrumb somewhere, like with a rule, or you just need to name them in a way that allows for, you know, greps and sort of other agentic search to find them.
But they purely loaded on demand, and they don't have any price to pay.
So you can have as many docs as you want.
If you have a thousand skills loaded, you might...
really cripple your agents today.
But if you have 1,000 docs that are just available, it's more about which docs that is loaded at the right time and can it actually find them.
So these are the three types we have today.
I'm sure we will have more types over time, again, all coming down to a reusable context so that the agent doesn't need to infer everything from intelligence every time.
Yeah, and I think the interesting thing there is something that we call activation.
I'm sure many others kind of refer to as well, which is, and this kind of like leans onto.
Back in the MCP days like it was 30 years ago.
When we were creating MCP servers whereby you need to be very clear in the name of the tools, the descriptions there so that the agent would use them at the right times.
It's very, very similar when we talk about those various skills.
You need to be very clear in the tools, in the skill name and in the skill description so that...
It doesn't just try to do things by itself just because it can.
It needs to use those skills at the appropriate times.
It just try and do things its own way.
So I think those are very, very important.
It's a good time to remember as well that while it's a standard format, they are not standard models.
And so the same instruction in the skill text right now would be loaded by different agents, by different models.
Although we know, for a fact, we have repeating data to show.
that the same words would not trigger haiku and sonnet and opus to the same action.
Opus is much more of a kind of smartass, right, and sort of can choose to say, no, I know better and won't do it.
You know, haiku might sort of need more sort of detailed instructions.
And so at the moment, skills don't really solve for that.
They are one standard unit of context.
But it's not sure that the same words will affect, will be the optimal ones for different agents.
So I'd sort of say, think of skills as an amazing new capability.
It is absolutely worth leaning into.
We're doing that in TESOL.
We think they're amazing.
They are maybe the most sort of standard way right now to reuse context.
But like MCP, I think they're also a piece of the puzzle.
And there will be all sorts of sort of tools or just sort of helpers that we would want to reuse for making agents successful.
That's really interesting.
So essentially the standard allows, it provides a kind of standard bolt fitting in the sense that you can integrate it with the agents, but your mileage may vary depending on which agent you use because it will use, it'll assess the wording and it'll use them differently depending on how, of course, they are implemented in the background.
So that's a good insight as well.
Okay, so skills.
We can use them as indie developers, as hobbyists, as open source developers.
We can also use it as in an organization to describe how we want to do certain things in our certain methodologies, in our certain processes that are key to our organizational requirements.
Now, when we think about adding skills as a first class citizen into our organizational process, our development process, what do we need to consider when...
building, owning those skills, and actually distributing it and expecting other professional developers in our organization to make the best use of that?
What do we need to consider?
Yeah, so I think it's a good question.
And I think kind of allure of skills is that they have this immediate impact on it.
You create some static markdown file, you do it with a create skill, very, very low lift.
And it helps.
Like right now, you immediately can anecdotally see that it works.
But I think just like with software, there are...
differences between kind of a one time it worked and therefore it's awesome and something that is a sort of long-lived assets that you now need to live with.
These are competencies that you want to reuse across the team.
So I think to kind of take skills to a professional level, to a team level, to an organization level.
you're better served by thinking of skills not as a markdown file, but as a unit of software.
This is a competency, a reusable competency that you want the agent to have.
So this sort of puts a different lens on...
on what are the tools that you need to be able to own skills, right?
And to be able to operate them and collaborate on them over time.
There are probably many things to handle, but I'd say the three to focus on is, one, you need to be able to test skills, just like software.
If you want it to remain working or even just sort of assess whether it works today, you have to think about what's correct and then test or, in the world of AI, evaluate against those.
Really thinking about how do you distribute that software.
We can talk about this more, but at the moment there's a kind of easy but sad reality that people are copying skills all over.
They're designed to be reusable, and yet we kind of copy and duplicate them and copy everywhere.
We've seen that movie, and we know where that ends, so that's not awesome.
And then the third is you have to think about how do you own them long term.
They will fall out of date, just like any docs or anything like that.
The models will change, and so a new model will come along.
It will think it's very...
smart, right?
Or you would want to use it with some sort of cheap model or open model.
And so we have to think about how do we maintain those skills?
How do we keep them up to date?
How do we allow a team to collaborate when the person that wrote them leaves the organization?
And so the kind of a whole life cycle management of it.
I think those are the core ones to think of.
So these are three key pieces to skills.
Evals, package management capabilities, and life cycles.
Now, It's really exciting that Tassel last week released support for skills and in fact would you believe it guy there there is coverage and functionality for evaluating skills, distributing skills through a package manager style mechanism, and also support for skills in a more life cycle, a complete life cycle way.
Can you believe that?
What are the chances of covering all three of this?
Thoughts become things.
Thoughts become things.
So we're really excited about that.
And you can go to tesla.io forward slash registry to learn more and see some of those skills in that registry.
It's kind of a good point also to just give the shout outs to the amazing Tesla team because I think this has been built.
superbly and quickly on it and couldn't be more proud of the team for pulling together excited by the capability possibly more excited even by just sort of the the sheer sort of power that was sort of seen in the team as we put it together yeah amazing let's go into these three um let's you know learn break them down a little bit let's learn about why we need them what they are and then let's talk a little bit about what tesla offers as part of this release start with evals there are a couple of different types of evals that we can offer here.
There's review evals and task evals.
Break them down a little bit for us, explain what they are, and then how does Tesla fit in this wonderful world of evals?
Perfect.
So evals are generally most broadly, a way to answer the question of how good is my skill.
Is it just like you wrote some words?
I mean, I think you'd easily understand that if you, whatever, put in unrelated texts or you rambled on for sort of an hour onto the LLM, it won't be good.
So it's easy to imagine something terrible.
And so there's a variety of stuff here.
Just be able to have a systematic way of being able to say, you know, how good is my skill?
And that's really important because like when we talk about the anecdotal approach, which you kind of like mentioned earlier in the episode, That's what we rely on today, which just isn't good enough.
Imagine if we were writing code and anecdotally this feels like it works, let's ship it.
It's just not good enough.
We wouldn't do it for any other piece of software that we consider a key part of our workflow.
So why would we do that with skills?
And we all know people that take forever to get to the point of it, maybe like I am right now.
So it's like, you know, when I was a child, I wanted this.
All of that type of context will not be very useful.
At the end of it, it's like, and therefore when you get commit, use uppercase letters.
That's not very useful.
So those are evils.
as a whole, and we might add additional ones.
We do have review evals and task evals, as you mentioned.
Let me actually kind of read out a little bit.
Review evals focus on just making sure that the skill adheres to the best practices that Anthropic has released.
Agent skills are cross-agent, but Anthropic is clearly leading the charge, and they created a bunch of best practices on their site that are a kind of good guidance on how to build it.
They're saying things like be concise, right?
They're saying concise is key, set appropriate degrees of freedom, you know, make sure you kind of have the right structure for the skill.
And so we, the review evaluations on it, just start by assessing it against those metrics.
That already has kind of given us some pretty interesting insights.
First of all, we've seen that if you use the anthropic kind of create skill skill, it indeed it scores quite highly.
Not that surprising.
And to be frank, we're using Claude to assess those on it.
So Claude, when it creates, uses those instructions, probably has a bunch of that input.
And so we're seeing, great job, Anthropic, if you're using a skill to create skills.
you're probably going to score well on all of these paths.
The other thing that we've seen is maybe a bit of a couple of points against the Anthropic, which is they might not be using their own create skill to build that.
And we've actually now run review evals on several hundreds, maybe even past 1,000 of different skills.
They include a bunch of the Anthropic ones.
And we've seen some in which they don't score terribly well on it.
I want to pick a little bit on the Anthropic's canvas design.
skill, which I guess as the name implies, kind of helps you design the canvas content.
And on the content side, we separate between our scoring for the description as a whole and for content.
It got fairly low scores, 27%.
And I just love the pros that Claude gives its own creators over here.
for this tile.
It says, conciseness one in four, extremely verbose with extensive repetition of concepts like craftsmanship and masterpiece throughout, contains philosophical padding and redundant explanations that Claude doesn't need.
Very straightforward there.
Like explaining what a design philosophy is multiple times, repeating the same principles in different sections.
So that's a good example of it.
And later on, it goes to the progressive disclosure one in four.
And it goes on to say, monolithic wall of text with no references to external files for detailed guidance, and so on and so forth.
I'm glad it's Claude slamming Anthropic, not us or anyone else.
It's all in the family.
And I'm sure this is a mission.
But I think what this goes to show is not at all a lack of skill in Anthropic and from the creators of Canvas design.
It's just the need for tooling, right?
It needs to be automated.
And it's possible that this was initially created with, you know.
say, they create skill, maybe it predates create skill, and then over time didn't get maintained.
So anyways, the review evaluations are about that.
They're about keeping you well structured and say, how are you applying the best practices?
So that's review it.
The task evals go a step further, and they actually create scenarios that you would run the agent against and see how well it did.
This is actually another one of the best practices that Anthropic.
It says create the evaluation scenarios ahead of actually creating the skill, like make sure you know what good looks like and capture those.
But few people actually do it.
It requires more effort on it.
And so we create, we have done this for tiles already.
We talked about that in some previous episodes.
For our documentation tiles, we create a bunch of context, but we create some coding scenarios and we run them with and without that documentation to make sure that documentation is good.
We do the same for skills.
I will say task evals for skills are more of a work in progress.
There's interesting questions about how do you best extract the evaluation scenarios out of the skill without leaking too much of the guidance itself into the task.
And so there's some work there.
So I'll share a couple of anecdotes of examples here, but take them with a grain of salt.
We've seen, maybe I've got two interesting examples of those.
One is we created task evals, 10 different coding scenarios, for, for instance, two fairly similar skills.
One is called Agent Browser from Vercel.
And we've seen that the skill was dramatically impactful.
It took the success rate of the agent as it tries to implement that from about 28% to 71% in success.
So very, very significant boost.
The skill is very valuable for Agent Browser.
And we saw another skill called Browser Use.
which conceptually is similar about the use of browsers within agents.
And with that one, we saw that the baseline average was 85%.
And in fact, using the skill was minimally negative.
It actually dropped that by about 3% each month.
And again, a lot of grains of salt, maybe we don't have the right tasks for it, but it's a good demonstration of maybe this skill is not that very helpful.
And we've seen a few of those examples.
We've seen cases where, for instance, some of the anthropic skills were not.
bad but were just not very helpful because they were just describing things the model already knows how to do which once again comes back to these right now are evaluations with sonnet maybe haiku needs it maybe opus doesn't need it and and you want to build those out um we'll have one more uh interesting anecdote which is uh someone codified uh carpathy's uh summary of like the best practices that it recommends for skills on it.
Someone put it together into a skill and published it, and we ran through it, and we actually found that it had, again, a minimally negative effect on agents.
We created 10 scenarios, so it did very well in some scenarios and worse in others.
One example where it did the worst, for instance, is that it just overcomplicated or overengineered a simple CSV-related task.
So all of those are, I think, great indications to...
to say, you need to know these answers.
You need to know whether your skill works or doesn't.
And you need to define what are the scenarios that simulate it.
And then you need to be able to test that over time as your skill evolves and as the models evolve or you're using other agents, et cetera.
There are so many juicy takeaways in this.
If you are a consumer of a skill, it's really important to know this data so that you can understand.
the impact that the skills that you're pulling in is going to make to your agent experience.
The second piece is as a producer of that skill, as the owner or creator of that skill, it's no use just creating a skill and assuming, again, through anecdotal evidence, saying, yeah, this feels like it's working better.
You need to, particularly with the scenarios, it's important to say, hey, do you know what?
This is making my experience.
This is going to make my chances of doing this thing the way I want it to be done.
three times, four times more likely to happen.
But I think the key thing here is when you mention the scenarios.
So if you have an eval where it runs across 10 scenarios, you can see what it's good at, what it's not good at.
And that provides you with the insight to say, these pieces of these scenarios are working really well.
I'm going to leave that description.
These other pieces, I need to change.
So from a producer point of view, from the creator and the owner of that skill, they can actually create the best skill.
From the consumer of the skill, they can then say, right, all these skills are available.
I'm going to pick these ones because I feel like they're going to be.
you know that they are showing the data is showing me that these can actually make it more impactful of course then everyone's environment is different it's that there is going to be somewhat an anecdotal yes this is actually working well or it's not working well but i think this gives us like the start we need yeah the second piece which really that you mentioned that i think is is so important is as models change and that variation across models or agents, there is going to be variety across the environments, as I say.
But even if something works for a model in my agent today, as that model grows, as the training data changes, my context needs to change as well.
My skills need to be updated, potentially, if it relies upon things that are in.
old training data or new training data, they will act differently.
And we need to use maybe even different levels of the same skill based on the agent I'm using or the model I'm using.
So these are kind of important considerations that we as users and creators of skills need to constantly be thinking about.
Absolutely.
And I will sort of say that there's a similar analogy to to tests in real-world scenarios, which is works on my machine versus something else is very much a real risk with skills as well.
As well as developers don't like writing tests.
Some love writing tests, but many do not.
And so people avoid this sort of hard question of what is correct.
And at TESOL, we will help you generate things, we'll help you build those out.
But over time, The definition of what is correct and what this skill should do is actually more important than the words that you use because the words themselves, they will change from environment to environment, from model to model.
Maybe over time, something that wasn't in the training data now is, and so you can shrink it.
You can actually say less now.
Maybe at some point you don't need the skill at all.
But all of those...
will be dynamic.
What actually would be less dynamic is what do you want the results to be.
That one you need to evolve it when your actual requirements change, your systems change, and you need to call an API differently, or your business goals have changed.
But I think not only are these evaluations critical for you to be able to make the most out of agents today and be able to not regress over time, the definition of of what good looks like is actually more important than the context itself.
Yeah.
I think the other thing that intrigued me with what you were saying is how essentially we're trying to create a set of questions from an answer that we already have.
We have a skill that defines the way we want to do things.
What we're trying to extract from that is, oh, okay, you're trying to do this.
You've got this end goal.
What I want to do is create a set of scenarios whereby we're not trying to feed or guide an agent into hitting that end goal as we would want it but more what is the general way that people would try and get to such a goal such an end point and then will it do it in the way that the skill is trying to so it's very hard to pull the question out of the answer almost yeah and evals are hard a good eval just like a good test They're hard to do well.
And I think it's important that we rely upon that good framework.
And that's what, of course, the team is, the AI engineering team at Tesla and others are working on.
And I think it's an important note that right now, again, this is all journey.
And at the moment, we are talking about creating evaluation scenarios for public skills, which are not ours, and with limited information because we just have the skill.
When we generate, for instance, documentation, we have so much more info.
We have the code itself.
We have potentially other colors.
So there's a lot more information to extract correctness.
But the true kind of correct evaluation scenarios come...
And that's what we do in Tesla when we sort of engage more with a full-on customer.
It comes when we actually get to kind of inspect reality.
Up front, you can extract a lot of that knowledge out of Git history and what have been the actions.
And over time, you can extract a lot of learning from agent logs and how we build.
And so all of those are correct.
When you are an organization, you're creating skills for yourself.
You're trying to optimize and tune them on an ongoing basis.
And so you get that sort of feedback data.
from actual agent users on it.
It's a little bit harder to do it for kind of third-party skills that we consume.
And we do hope, just like with documentation, we today are creating evaluation for someone else's skills, but we encourage and would love the owners of those skills to come to us and we will pass those evaluations and the ownership of the eval scenarios to those people so they can, you know, kind of tune them and correct them.
And then we'll just, you know, run the evals and publish them.
I love it.
And just...
clicking around actually on the TESOL.io slash registry where you can see the tiles and the skills and there are a bunch of feature tiles, but also then you see the top performing skills and tiles.
And what's great is you can see, you know, I can click through and I can see the skills and I can see the eval results on various skills as well with different scenarios that I can click, you know, I can open up the scenarios, I can see what the scenario is doing and I can see the kind of like, you know, the with and without the skill, with and without TESOL and you can see what is good and what is not.
And if you dig in a little bit, you'll also see see some of the prose.
You see some of the text that Claude writes as it assesses it.
Yes, it's entertaining, and it's probably reason enough to keep it on the website.
But it actually has some very concrete advice to say, hey, this is the problem.
Like that monolithic wall of text comment was entertaining.
But it's also very clear guidance to the owner of that skill to say, hey, you should break this up.
We should totally have modes of how we want Claude to give us that information.
So if it's saying, OK, this is how this is the review eval you can get it to be nice to me I'm fragile be nice to me you can get it to a little bit of it yeah I'm in a tough spot well I think the other way is like the glimmer of hope so like there will shortly you know we do this with customers today there will be an optimized button like Hold out hope.
We have this.
Do you want us to fix this for you?
Click this button and we'll create a new version of it.
Which is what LLMs and things like that are good at as well.
It comes back to you need to define the evaluation that is correct.
Because if we generate documentation with an LLM and we generate evaluation with an LLM and the two are misaligned, you have to choose who do you believe.
And what we want to believe is the evaluation.
We need the owner to bless that.
Once they do, it's actually not that complicated to create agentic pipelines to optimize the context to achieve the goal.
We covered evals quite a lot there.
I think evals is actually a really important piece, though.
So I'm really pleased we kind of went into depth on that.
The two other pieces that you mentioned, package management or package manager capabilities and also the lifecycle of skills.
Let's talk about those, starting off with package manager.
Yeah.
So I think there's more of a technical element to it.
really comes out of thinking about skills as software, not as documentation.
So today, when you want to install a skill, first your option was to just sort of go off and copy it from wherever it is that you want.
Or you created a skill, put it somewhere so your team can do it.
And remember that skills are designed to be a reusable entity.
So really, the core element to them is spread them to your team.
That's inconvenient.
We've seen a really handy tool from Vercel called skills.sh.
And so today you see within a week it took it up and people do NPX skills I and provide a GitHub repo.
And what it does is it copies something from a GitHub repo to their .skills folder.
And now they...
kind of committed to their repo, right?
And they duplicate it.
And what happens if the GitHub repo got an update?
Well, they're none the wiser.
They don't get that.
We've even seen worse than that.
We've seen the fact that, you know, because people install skills, but then maybe other people in the team are using different agents, they might even duplicate that skill into both the .cloud slash skills and the .cursor slash skills.
We're seeing an entertaining...
effect of all these different agents going and reading skills out of .claw skills.
It's really a mess.
And that is just not kind of a good recipe for a happy future.
We need something better.
And this is a problem that we've already solved in software.
We know how to handle.
reuse in software.
We call it a package manager, right, or a dependency system.
And it requires a few things.
It requires, first of all, like some form of manifest file that knows what you've downloaded.
Actually, even before that, you need whatever piece of content that you have to have a version, to have an identifier, to say, well, this is the thing that I've downloaded, and maybe Semver-style indication of...
of the delta and ordering of those versions.
Once you have a manifest file with those skills that you have, you should be able to run an update, run an install, maybe add some other criteria, separate ones from different environments like dev and production.
So all of those things are solved problems.
We have a lot of practices that we can bring to fore to make them work well.
We also know that pretty much every programming language has its own.
dependency system because each of them have some subtle important differences in how you want to handle the dependency relationships and installations and environments.
And so what we need is we need a package manager for skills.
And I will sort of say skills and context because, as we mentioned, skills is not the only type of context.
And that's basically what we're building in TESOL.
We've had the TESOL registry for a long time.
It's always been the sort of concept of, you know, this is a...
context that is version, that is consumable in a way that is like a professional software development team tool.
And today we extended that to skills, which is really, really exciting.
So it allows you to TESOL skill install whatever skill that you want.
We're not going to get in your way if you just want to download something off a repo.
But without any further effort, we will remember that you downloaded that in a manifest file.
and will allow you to update it over time.
We will install it to whatever agent that you want, and by default, we will still commit it to your repo, so we don't get in the way.
Maybe other team members have not installed TESL yet, but we think the right destination for you is to change that mode and stop vendoring those dependencies, stop committing them and duplicating them, and having just the manifest file.
And just like you npm install or whatever your equivalent is in your repo, You also Tesla install or Tesla skill install if it's just the skills that you want, and it will put them in the right agent for the right user for what they're using.
And so we generally believe, and again, it doesn't feel like a complicated or sort of a harsh statement, is that skills...
just sort of need a better distribution system.
Yeah, and I love the two differences there.
Obviously, what we talked about with some solutions is more of a directory, I guess.
It doesn't necessarily host various skills.
It more points to other skills.
There are benefits and drawbacks to that approach.
I think when you have something more like a traditional package manager where it kind of...
hosts that content it can do it can it can look after a bunch of those things include a bunch of things for that content including the versioning and so forth um and including in the manifest that says these are the things that i want to use um so i think i think that ability to say okay if i want to you know push something to tesla publish a skill to tesla i can have all of that cool stuff including things like evals including things like the versioning and things like that however if i just want to use it lightly I could actually say, actually, I want to point to this GitHub repo.
I don't want to necessarily publish anything or own anything within the test or registry.
And so that will still happen.
We'll still add that to manifest.
But of course, there will be a few little drawbacks.
We wouldn't have the evals.
We wouldn't have things like the ability to pick a version because it's just there on GitHub.
It's not necessarily a version in a registry.
So we can't identify.
more points to the to the latest but these yeah i think these are really really important things when we think about skills as a first class citizen in a developer workflow these are the capabilities that we need to to do professional development to do You know, auditing as well.
What skill did I use to create these types of things?
Well, you actually have a look at your manifest and you can say, well, this was the skill I used.
These were the processes I used.
And I think I don't want to diss the existing capabilities.
I mean, JavaScript thrived before NPM.
You know, Java thrived before Maven.
They're not necessary to be able to provide value.
But just at the pace that everything is happening right now, I think.
I think we need to move quickly with this as well.
And over time also, it's fair to expect TESOL to introduce other things that you have in package managers, like can we inspect skills to see if they're malicious?
Can we already run these evaluations to sort of say, give the consumer some indication to say, well, I'm seeing 17 different skills that are trying to do the same.
Give me some indicators to tell me which one might be better for me, what worse.
And so all of those traits will come along.
learning from the past versus trying to ignore it.
It's a similar pattern to other things that we've used, so why wouldn't we do things in that way?
It's hopefully a healthy ecosystem.
So a lot of the stuff that we've talked about from the value that a package manager style solution offers actually leans into a little bit of the requirements for what we need for software in a lifecycle.
Tell us about what you see here.
the needs on a skills style approach with lifecycle and professional development, professional workflows as a constraint.
Yeah, for sure.
And I guess, again, they all sort of stem out of the, you know, once you think about skills as innovative software, the list that you need for them becomes almost obvious.
Skills need to evolve.
And so, first of all, Again, you need to version them or something, you need to commit their source material, fine, we have that.
But then when you collaborate on them, you need to be able to know that a change hasn't regressed.
So that's where evals kick in.
We talked about them sufficiently, you know, but you need a system that every time there is a pull request or modification.
it can modify it.
Skills rarely live in isolation.
They are a description of a practice, but oftentimes they describe a system or a library or even the practice itself evolves.
And so you need some ability to keep them fresh.
So something where every time you modify not the skill, but you modify an API next to it or something else, you need to be able to track and know that you need to update the skill or not.
And again, sort of make sure that it hasn't regressed.
Those are the core, as you build it, you need to define it.
They need a standard CD system.
If we talked before about the Git style and the CI system, you need some element of deployment.
to say, okay, publish this now to the registry or commit it to the relevant repos if they're changing so that those things are automated and they're not overlooked or create manual toil.
And lastly, you have to think about observability.
I mean, evals and synthetic tests are all good and well, but they don't represent production, right?
They don't represent what is deployed.
And so you have to think about how do you observe skills in production, which is basically...
observing agent logs and be able to extract out of those cases where the skill didn't work and the user had to either manually invoke it or had to correct its behavior, cases where failures repeated and it might be an opportunity to create a new skill.
So all of those represented this sort of full system, right?
editing and collaborating on the code of skills on it, to a CI-CD system on it, to observability and closing that loop.
I think you need all of those for skills.
And our vision at TESOL has always been that software development will go from being code-centric to being spec-centric, right, or intent-centric, where what development teams do is they collaborate on intent, on capabilities, on how to build software and what it is that you build.
And I think skills are a great manifestation of that.
And so we need the software development paradigm around this.
We're going to expand into debugging and others.
So we're very excited about that.
A version of this is what we've been building at TESL all along.
And we work with some forward-thinking customers that are building and developing context this way all along, long before skills got introduced.
And they've been amazing, and we're learning from them.
And so you can expect from us more and more capabilities, both self-serve.
We definitely want to help open source skilled developers be able to maintain and do all of that with minimal toil.
And of course, for organizations.
So you'll see more coming on the self-serve side, and we'll sort of announce it as it comes.
But if you can't wait on it, send us a note at contact.sl.io.
And we'd love to work with you.
And truly excited about it.
kind of the point we are in the timeline in the sort of creation of indeed this sort of intent centric software development paradigm.
Yeah, super exciting.
And of course, if you're interested in seeing some of the evals, using some of the skills on our registry, wanting to publish skills on our registry, head to tesla.io forward slash registry.
And of course, you can see all the data there.
And of course, you can start getting involved, grab the CLI and do all the things I just said, publish, consume and so forth.
And I will note.
We love feedback.
We appreciate feedback.
Please send any thoughts that you have.
And we did add core capabilities.
We didn't proactively run task evaluations on many thousands of skills.
We don't feel like we're quite ready to do that yet.
But if you see a skill that is missing or one that you want us to evaluate, there's an easy button over there and we will quickly get on it.
Awesome.
This has been a fun conversation on skills and I think it's an important one for people to learn right now.
You might eventually acquire some even.
We'll see.
We'll see.
Look for a few.
Let's not over promise.
Let's over deliver.
Awesome.
Thank you very much, Guy.
Always a pleasure.
And thanks everyone for listening.
Tune in next time.