# DataCamp AI Tutor Economics and Architecture

**Podcast:** Chain of Thought | AI Agents, Infrastructure & Engineering
**Published:** 2026-11-14

## Transcript

easily 10 to 20 times higher than our previous kind of costs, we would have made the business worse.
The cost will go down significantly and it's a huge area of focus as well.
The cheerleader is essentially a system of several agents that collaborate.
But for us, we have to respond in milliseconds.
Welcome to Chain of Thought.
I am your host, Conor Bronsden.
For more conversations with the people building AI systems, such as Jonathan Cornelison, DataCamp's co-founder and CEO, and Yusuf Sabar, their chief AI officer and the founder of Optima, the learning company that DataCamp recently acquired.
They have been working through how to make personalized tutoring accessible to people around the world and exceedingly useful.
how they're building DataCamp, what they're doing with tutoring through AI, and much more.
Let's set some groundwork before we dive too deep into the conversation.
Yusuf, when a learner is going through on DataCamp and something goes wrong, let's say they fail a test and they're having problems.
What do they do when they pick up the Datacamp platform?
And what's their experience as they start to interact with it in a way where they need to improve?
Really, very good place to start.
So if we begin a tiny step before that, you think of what the TURS is trying to do versus what we call static content, so lessons or videos that have been pre-generated.
Essentially, we're trying to solve two problems.
The first one is, we are very different.
We have different interests, we have different goals.
So even if we are learning the exact same material, that material would make a lot more sense if it's presented as a solution to our very different problems.
That's one.
And the second is we have different backgrounds, we have different gaps in knowledge.
We sometimes need different amounts of time to understand the same concepts.
So the tutor can adapt the flow of the lesson, what gets covered, what gets skipped.
based on what the person knows.
And to make that work, it can't just have a lesson or a video that was pre-created, it kind of creates that as it goes from a well-defined set of inputs so that it's not making up the information, every fact in the lesson, every asset.
Kind of like a good teacher wouldn't let the students sit down as he tries or she tries to create slides.
They come in with their figures pre-created.
They've really thought through their material, but they adapt the delivery.
That's exactly what the teacher does.
So at a high level, we think of a great teacher as someone who exhibits a set of behaviors.
Those are the behaviors that come together to make someone an excellent teacher.
And one of those behaviors is they're very good at giving feedback.
So they're thoughtful about how they go about giving feedback to their learners on whatever it is they're working on.
So the way the tutor works is we take those behaviors and we try to recreate them in AI form.
So this is an example, and it's the behavior that guides the AI tutor on how to deliver feedback to a learner on whatever they're working on.
So very high level.
First, when the learner submits something, it would check where the criteria for success met.
If that's true, then it will celebrate their success.
And that celebration is kind of proportional to the amount of struggle and effort that they put into the work.
And it's focused on that as opposed to like their innate talent.
And then it will focus on recapping what was learned from that struggle.
So sometimes we find learners who just try it through trial and error get to solve a problem, but they really don't pause and reflect what did I learn out of this.
So it ensures...
that whatever learning was captured and clear.
But what if the criteria are not met?
So it will see, did they actually finish solving the problem?
Or they are still halfway through it and they just want to get some feedback.
They're very different things.
And prior to AI, you'll submit and say it's wrong.
It's not wrong.
I'm still working through it.
I just need some initial feedback on the direction.
So it will figure out, okay, this is the current state.
They're still starting.
They want feedback on.
So it will appreciate the progress and let them know that they're on track or off.
but it will not preface them for not having a complete solution yet.
Then if the solution is actually, like there's an actual problem, which is the case you asked about, then first it will check, does it understand where that is coming from?
Is it like a misconception, a typo mistake they made while they're just writing quickly?
Like if it believe that the source of this error is a misunderstanding, then it will clarify that and ask them a question.
instead of giving them the actual solution.
But if it doesn't know what the cause is, like it's not about, like, I know what the problem is, but where is that coming from?
It'll actually ask them.
It'll say, walk me through how you got here, just to try and understand where the error is coming from.
With one set of errors, usually there are not one problem, but multiple.
It will focus on one.
We found through our experiments, when we give the learner, hey, your solution is wrong for these six different things, they break.
like most learners, they just don't continue.
It's just too overwhelming.
So we give them the main issue that will get them to move one step forward.
And then the second set of issues are shared.
And the processes, if on one particular issue that they're kind of trying to work through, there will be a series of hints, phrased mostly as questions and increasingly more informative.
So the first ones, we'll try not to give away any detail to let them figure out on their own.
But eventually, It's clear that they can't move forward.
So in the past, we used to share the solution, but then we stopped doing that.
We actually asked them first.
We gave them that agency.
They make the decision that they want to see the solution.
We have found not many, but quite a sizable population of learners who would actually want to try 20 times, 30 times until they figure it out, and no one will deprive them of that.
So that either would ask, it seems we are stuck on this problem, or we're not...
There's something that we need to clarify together.
Would you like to share the solution?
And then we discuss it.
So if the learner says yes, then it will share the solution.
If they say no, then it will respect that and let them keep trying.
So that's the eye level how that behavior unfolds.
Before we dive in, I want to say a quick thank you to our presenting sponsors for Chain of Thought.
Ingest for long-running agents and Walrus Memory providing portable agent memory.
Plus, this episode's other sponsors, Coralbrex.ai, providing lightning-fast open model inference, and Wisdom AI, the enterprise agent analytics platform.
Jonathan, I'm curious from your perspective, how have you seen user behavior shift on the platform over time as you've implemented the AI tutor and expanded this program?
Very good question.
So when we kind of embarked on this journey, obviously there was a lot of excitement because our vision was always, We want to make sure people learn by doing it, and we ultimately want to get the experience to be as close as possible to an actual tutor should we flip the switch.
So we actually started by doing a clean A-B test where we did a bake-off between the AI tutor experience and what we call the Datacamp original or the Datacamp classic experience.
And what was very exciting to see is that Even this early version of the AI tutor, the version that existed six to nine months ago, it already performed better on several important metrics.
We have a freemium model on the consumer side.
And so ultimately the willingness to pay is a key indicator of how well people are kind of perceiving the experience.
And so willingness to pay was...
immediately higher, engagement was higher, satisfaction was higher compared to the Datacamp Classic experience.
So overall, it was kind of a very clear win.
Even though I feel like we're at the very beginning of building these amazing AI tutor experiences and based on all the data we're constantly gathering, based on improvements in models, improvements in different parts of the technology, I think the this is kind of the worst tutor that we'll ever have.
And so I'm very bullish on the fact that we're at the start of this S-curve and that there's a lot more to come.
I mean, I'll say it makes me personally excited for the next time I am actively seeking to take a course and learn something because it's so clear that the technology around it is changing.
And if you can access this type of tutoring...
As you put it, Joe, there is such an opportunity to learn better.
Yusuf, right now, is this all running off of a frontier model or what's the approach actually underpinning these decisions for the tutor?
The tutor is essentially a system of several agents that collaborate.
So the kind of experience that's put out to the learner is this coherent single entity, but it's actually a set of entities that collaborate.
to deliver that experience.
And there is at least four different models that underpin those different entities.
So the core tier that does about 70% of the work is a frontier model.
It's a Sonnet.
And we just upgraded it to 5.5 earlier today.
And it was a good step up.
Not a big step, but a decent step in...
outperformed over our evaluation.
So far we have found Sonnet to be the sweet spot for the tutor.
So smaller models, you saw one example of a behavior.
There are at least 30 core behaviors and another 50 secondary behaviors.
That's a massive amount of instruction that the model needs to stay coherent about.
And with a smaller model we found that it just loses that coherence.
This is a very long time horizon task.
The tutor will run a lesson that can go anywhere from 15 minutes to an hour.
It has to stay coherent all through that.
We do a lot of work to keep the context within limit, but it also has to remember, if someone is submitting a solution, it has to say, oh, you're trying the same thing we tried earlier.
Why don't we think of something different?
So if it doesn't remember what was tried earlier, it just can't show that coherence across the lesson, even across multiple lessons.
So the board needs to be able to handle larger volume.
But at the same time, it needs to be fast and the cost needs to be reasonable.
The slide has always been the tier that is just right for this application.
But recently, we started to see the lower tier.
So I think the Haiku slash Luna tier becoming capable enough that we can actually use it more effectively for a more substantial portion of the workload.
Before we used to use models like Gemini Flash, for quite a bit of the work.
It's exceptionally fast, mostly because we are hosted, so there's just almost no latency.
So for applications that require very high speed, usually simpler cases, we have relied on that.
Maybe it does about 5 to 7% of the work.
But the exciting thing for us now is this, what used to be the lowest tier of models.
is kind of stepping up that it can actually do some of the heavy lifting that was just out of reach for them in the past.
This episode is also brought to you by Coralbricks.
Coralbricks serves open models through one API at the lowest prices and the fastest speeds.
DeepSeq v4.1 Flash streams at over 300 tokens a second.
Plus, Coralbricks also just put out Motionbench, a benchmark that asks a simple question.
Can a model animate?
I've personally contributed to the open source MotionBench and was impressed that DeepSeq v4.1 Flash managed a close third place behind a leading Cloud Opus 5.5 at about an eighth the cost per task.
The animation for this ad came out of that benchmark.
We used Cloud Opus 5.5 and DeepSeq v4.1 Flash via Coralbricks to create and edit it.
Get lightning-fast open model inference at chainofthought.show slash coralbricks with a free $5 in credits on signup.
plus an additional $5 credit on first purchase.
I assume scaling this has been a challenge given inference costs and the architecture involved.
Jonathan, how have you been approaching the scaling challenges here?
Yeah, so maybe as additional context on the previous point, like cost is a huge consideration, but one of the limiting factors is speed as well.
Like people like really good teachers, but...
If they take five to 10 seconds to answer, people don't care.
They will be gone.
So the speed really matters and puts a whole constraint.
But then from a business perspective, cost really matters as well.
For context, we had over 10 million hours of learning on the platform last.
We've scaled to tutor from kind of almost nothing nine months ago to now we have a run rate of over a million hours.
of learning per year.
So it's grown quite rapidly.
And the cost initially was measured in many dollars per hour of learning.
And so it's a real business challenge because that's easily 10 to 20 times higher than our previous kind of cost setup.
And so even though a lot of our customers, a lot of our clients really liked the experience.
it was essentially too expensive to scale.
And so that's been a huge consideration in terms of how we approach this.
It's also one of the reasons we fundamentally believe in open source and open source's ability to get costs down.
But it's good to see even at the frontier labs, like there's more competition now.
So there is more price pressure that we can take advantage of as well.
Yeah, I'm curious if you've tried like DeepSeq v1 Flash or any of the newer open source front-air models and how that has impacted decision-making here.
Yusuf, you wanted to speak to that?
Yeah, happy to.
So we have this ongoing experimentation stream where whenever an open model kind of comes out, we immediately run it through the evaluations.
We have them running on...
like an internal system where we can choose any of the models, run lessons with it for quick, like to get a kind of quick feel for how it's performing, where it's failing.
So both tests like evaluations, which are quite accurate, but that feel for is like, there's this, you know, the spirit of the model, which varies quite a bit and can't be captured easily in evaluations.
So we run them through both.
And what we found is usually many of the open models are disqualified, like just still on arrival, like right off the bat.
At first, it's usually speed.
So quite a few of the models, even running on proper hardware, they are very slow.
So they take quite a bit of time to respond.
And what we found is learners, if for a lot of agentic applications you have, you can take minutes to work on a problem.
But for us, we have to respond in, milliseconds.
So speed is critical, otherwise learners just lose interest and their attention drifts.
So that millisecond kind of time window makes it quite a challenge for many of the open models.
The second issue that we've found is getting proper hosting is not easy.
So you could, you know, for experiments or for some applications, you could run quite a few of them.
But getting hosting that meets all our criteria, is often a bit tricky.
Even with Google's own models getting them hosted properly on GCP, sometimes there isn't enough supply of GPUs that are liable.
Sometimes it ends up being more expensive, like once we get the hosting setup right.
Because to get the same throughput, it ends up being a bit more expensive.
And you need a supplier who has built a really robust caching architecture.
Like what you get from the major labs, you get this incredible caching architecture, which allows you to run frontier models at a 98% discount, which is not as developed, I'd say, for some of the open model options out there.
But I think every month we are closer.
Very recently, just a few weeks back, a bit more than a few weeks back when the Gemma 4 class of models was generated.
Actually one of those models did really really well, actually outperformed our best performer at the time.
But most of the challenges to getting it run and scale were just getting reliable hosting.
Most of the options we found just didn't have enough capacity for what we were asking for or had enough capacity but latency was a serious issue.
or didn't have the right caching infrastructure.
So we end up saving significantly over what we are paying at present.
Interesting.
Jonathan, talk to me a bit more about the economics here.
So if you are seeing great savings from some of the open models, but you're also having challenges reaching the goals you need, is that more about optimizing for, you know, per learning hour?
Is it about how to get learners to mastery?
What's kind of the unit here you're looking at?
I think from a business perspective, it's actually quite simple.
From a business perspective, what I care about is kind of the cost per hour of learning.
An hour of learning is generally a proxy for value deliverance from a customer's point of view.
And that costs I think eventually will be similar to kind of our kind of server costs to run our interactive learning experiences.
I think that that trend seems clear.
The only question is, do we get there in four to eight weeks or does it take us another year?
But the trend is very clear.
And that's also the bet we're making.
At the current costs, we don't have negative margins, but our margins are not super interesting.
I would say, on kind of the AI tutor experience, even though we deliver a better product to our customers from a kind of financial point of view, we would have made the business worse.
But I'm very confident based on everything I'm seeing that the cost will go down significantly and it's a huge area of focus as well.
And I think eventually will be and become kind of a competitive advantage as well.
If you can have a really excellent tutor, but you can deliver that at a lower cost per hour of learning, that will be kind of an important competitive dynamic in my mind.
But Connor, you implicitly refer to another angle to look at this as like, well, is it true that one hour of learning is the right metric to kind of think about value?
And this is actually an area where...
switching even from interactive learning experiences to the AI tutor gives you a very different result.
So for context, if you were to take a course in Datacamp, Datacamp Classic course, you would have a range, maybe a three-hour course.
Some people would finish it in two hours.
Some people would finish it in four hours.
But most people would be around three hours, right?
And Datacamp already had a range where if you...
If you think about a video course, like the classic Coursera type experience, a three-hour course is a three-hour course.
Now, fast forward to working with a tutor, you actually get a much, much wider distribution because guess what?
Some people actually can get through this material and can achieve the learning objectives in maybe in half an hour.
And some people will take six hours.
And it depends on many things, like how much background they have, maybe...
how much they know about similar topics, how fast they're kind of able to learn.
And so this is kind of a new way of thinking about value delivery because if I'm an organization, I want my best people to spend, to learn the skills they need to learn, but to spend as little time as possible.
But at the same time, if I have a group of people that needs more time, I actually want them to have the skills.
I don't want them just to take the three-hour course and then half of them still don't really know.
And they still don't have the skills.
Jonathan, as you think about the goals for DataCamp going forward and for building the best AI tutor that you can, what will GREAT look like?
for you internally and how is that going to align with the goals of businesses that are sending their team members through Datacamp?
Great question.
So there's really two parts to the system we're building and the system we're envisioning.
There's the AI tutor itself, which we feel is already in a very good place and is kind of leading in the markets.
So there's internal improvements that make it a better teacher, a more effective teacher, a more efficient teacher.
That will be hard to immediately see, but you'll kind of know when you go through it.
But then there's things like modalities.
I'm very excited.
There will be a time where there will be an audio-only version of the tutor where you can go back and forth.
on your commute on a mobile device, you can kind of have this private tutoring session just kind of going back and forth with your tutor on the phone, which I think for a lot of people who are busy or who are on the road in commercial functions will be a great experience.
But that's all the AI tutor system itself, where I think this is going to be a real game changer.
And this is a system that's much, much earlier stage.
We're also developing technology to prepare the material for the AI tutor.
We internally call it the AI creator.
And that will eventually enable us to not just teach data and AI skills, but to teach any professional skill and to have a system that ingests existing knowledge, could be their own internal company knowledge, could be a really good book, a really good YouTube video.
And then kind of transform that into something that the tutor can really teach.
And that's where I'm very excited because once you have the whole system, you kind of have an omnipotent tutor that can kind of teach anything.
So that's what we're kind of working towards.
Obviously, it's going to take many steps to get there.
But we're very excited about that kind of future that we're building towards.
Season 4 of Chain of Thought is presented by Walrus Memory, the portable memory layer for AI agents.
Your agent has learned your code base and how you work.
Now you want to use that context in another tool.
Exporting a file gives you a snapshot, but what happens when the context changes?
Walrus Memory is a portable memory layer that lets your agent store context for later use across tools.
Python and TypeScript SDKs plus native MCP support let you connect it to your agents no matter where they are.
You said who can read and write, so sharing context doesn't mean opening up your whole memory store.
If you are building across tools or model families, take a look at Walrus Memory.
Learn more at walrus.xyz slash C-O-T.
That's walrus.xyz slash C-O-T.
And I assume one of the challenges you face around this is latency.
Yusuf, you took us through a bit of the architecture earlier.
How do you think about latency challenges?
with the tutor as you obviously were have to work with learners around the world.
So we found that latency is extremely important to success of the tutor and of the learners.
So even if it delivers the correct response, the fact that it needed several seconds to get to that response can often really subtract from its value.
So what we try to do is we have this a set of limits on how long we can allow it to take in terms of time at every kind of interaction.
Some interactions, it needs to be really quick.
Some interactions, it's generally a difficult problem that it has to solve.
And a wrong solution to the problem can derail the lesson, so we can allow it to have slightly more time.
Like there's about 6% roughly of the responses during a typical lesson where we have to turn on thinking.
So it needs to make a critical decision.
Like one of those critical decisions is it's about to start teaching something and then the learner says something.
They ask a question or they share something that implies that they probably know this material.
So it's a very important judgment call.
There's this statement, a good human teacher would just quickly catch on this, oh, you said that and you probably know this, and then they wouldn't just walk them through it as if they're hearing it for the first time.
That's a kind of a critical judgment call.
If it gets it wrong, it can really, you know, take the lesson in a direction that would take, you know, several minutes to recover from.
And at those moments, we kind of allow it more time to think before responding.
Whereas in some other cases, it's just a continuation of an idea that's being delivered.
There's no new question.
So those moments, there is a lot less time.
The other thing is we have this massive asymmetry in how the system works.
So once the learner responds or asks a question, they want an immediate reply.
But then they take a very long time to think about that reply, to do work on an exercise, and then respond.
So we try and use that downtime to do a lot of the preparation.
If you look at a lot of the common...
like AI engineering architectures, they rely on like tool calling.
So you set up a bunch of tools and you allow the agent to kind of call tools.
For tool calling, you immediately add a round trip.
So it has to decide if it's the right tool, it has to generate a response, wait for the tool to return, and then generate a second response.
So for us, this is almost always off the table, but we carry something that is essentially the benefits of tool calling, but where most of the work happens during the downtime.
where the learner is processing the previous response.
But we have to kind of make several guesses like is the learner likely to answer in the positive or the negative, likely to get the response right or get it wrong.
And because where the lesson goes from that point can be different and we have to prepare like some material that doesn't get used because the lesson went a different way.
A lot of the work we do is kind of optimize that prediction.
So the chances that we throw away tokens we generated is very low.
And through that we can also make the response really fast.
Even audio, you see it almost starts speaking immediately because part of that audio has also been pre-generated during that downtime.
But we could end up not using that because the learner said something that took the lesson in a different direction.
That's why we try to optimize that to get latency really, really low by kind of taking a bit of risk on where the lesson might go next.
It's super interesting to think about this risk idea.
Are you able to use caching across different learners to kind of reuse specific responses that maybe didn't get picked up by one learner?
You can apply it at scale across other learners, or how do you think about avoiding some of this potential tax if a learner goes a different direction?
So, think about there is a sequence or there's a plan that's pre-prepared for the flow of the lesson.
And that plan, we keep optimizing it based on real lessons and how they actually unfold.
So, the plan, if that plan has been well thought and it's been optimized through actual trajectories of how lessons were taken, then it's less likely that the learner would need to go off plan.
Now, so usually when going offline, it's more often we're skipping part of the plan as opposed to coming up with an entirely new plan.
Because the objectives of the lesson are fairly kind of preset in advance.
The other part is, a lot of the time, what can be shared between lessons is a lot of like the infrastructure.
So a lot of the learning plan itself across the different lessons.
We can, like you can do the same lesson.
We're using, let's say, a lesson on SQL, for instance, a data set about a podcasting platform with data that's podcasters and shows and so on.
And doing that in a learning platform.
And all of the data I'm seeing is courses and students.
So the example scenarios that are used to run the two different lessons are very different.
The material is the same.
So we try to have a good...
kind of subpopulation of learners that is in each of those groups.
So there is a variant of the lesson that's cached through some of the other learners.
And you know, like a lot of the power comes from like deciding where to put different parts of the prompt.
So as much of it as possible is common across the largest possible base.
So we spend a lot of time on that.
How can we get to, you know, 90%, 94, 95% cache hit rate?
as opposed to something lower.
We found that a better model allows us to take liberties with getting the caching to the mid to high 90s that a weaker model just can't handle.
I've got a lot of the tricks that we kind of do to help a weaker model involve dynamically altering the session.
The way it would unfold it is just too much for it to handle.
So we remove previous responses, we hide certain things, we summarize certain parts.
But that means those parts are no longer cashable.
So the price could grow higher for those reasons.
Season four of Chain of Thought is presented by Ingest.
Long-running agents need endurance.
They must endure waiting for users and weather tool failures.
They must take each new jump in model intelligence in stride.
Ingest makes long-running agents durable, observable, and improvable over time.
Check out their generous free tier and a sleek local dev server for testing your workflows.
Grab the ingest link in the description or visit ingest.link slash C-O-T dash pod to get started.
Build for the long run with ingest.
And as you make these architectural decisions around latency, caching, reliability.
Obviously, you still have to make sure this is actually serving your end user.
Jonathan, what are you doing to test that a tutor is actually helping someone learn rather than simply producing an answer the learner may like?
I think the high level answer is this is actually quite well established in kind of scientific literature.
Even 40, 50 years ago, you had this researcher called Bloom who essentially kind of show that conventional classroom education versus one-on-one tutoring yields very different results in terms of educational outcomes.
So you can move someone from kind of an average performance as a student, you can move them into the 95th percentile by kind of having one-on-one tutoring.
And if you look at the real world, this is a common thing, right?
Like if people can afford it, they will often get...
private tutors for their children.
And that's for very good reasons.
Because if you think about what a good tutor does is they adjust the base, they kind of adjust all the examples to the kind of world that learner lives in.
They will catch any misconceptions really fast.
They will challenge them, but not too much.
So there's so many things a really great tutor does that We essentially have encoded into different behaviors and that we have a whole eval system to kind of check, is the tutor actually demonstrating these behaviors in a reliable way?
And that's a continuous improvement effort where we monitor, obviously, all the interactions students have with the tutor and some of the friction points between a learner and the tutor may then give us ideas around.
how can we further improve things?
Now, Connor, all of that being said, like this is not yet scientific proof that our tutor is better, which is obviously what we want to get to.
I was actually reviewing the scientific literature a little bit last night.
And so there is research that I think one in Nature from last year that already kind of researched AI tutors that have this kind of skeleton around that are demonstrated to be better than conventional kind of teaching approaches.
Just the LLMs without the scaffolding, I think even there's research that indicates that's not necessarily helping learners progress their kind of understanding and their skill level.
It may make them do better at the test, but that doesn't mean there's more learning.
that happens.
But long story short, I can't wait for the day where we have scientific collaboration and we can actually show you some hard data.
For now, it's a collection of the existing scientific literature and then actually looking at the data that I have a very high degree of confidence that it already is better and especially more effective where In some cases, you only need 30 minutes to get people there.
In some cases, you may need an extra hour.
But the end point that those two learners reach is much more similar versus if you put someone behind a laptop and say, watch this video for half an hour, you have no idea.
Like, were they even paying attention?
Have they absorbed the skills?
Have they not?
Like, you have no idea.
And unfortunately, a lot of corporate learning that...
That's still what it is.
You say, oh, people watch these videos.
Great, we've got a check mark and we're done.
And so I feel very confident that we're already beating that.
But there will be a lot of research on this topic in the next few years.
And we definitely want to, if anyone listening is on the research or on the academic side, we're very open to collaborate.
and find ways to share data.
That's a great call out.
I will say anyone who is listening to the show and wants to reach out, drop a comment or reach out to me and I'm happy to put you in touch with Jonathan Youssef because they are doing some very interesting things here on the front of learning.
And I think especially as we shift our day-to-day for work to be one where so much of execution is happening via the AI model where A chat window is our interface with a lot of work that is to be done.
We are going to have to rethink how learning happens and be very structured about it.
Because right now it is so reliant on individuals to keep their skills from atrophying as we move away from execution.
And I think a really interesting philosophical thing that we have talked a bit about in the show already, particularly in our recent episode with Larry Voss.
is the gap that has opened up for early career entrants, where it used to be that if I was joining a field, I would be given a lot of the grunt work in order to learn.
Because look, yeah, my leadership, my mentors within the industry didn't necessarily want to be taking on that grunt work themselves.
They'd done their time.
But now they can hand it off to an AI model that is often doing it more efficiently, that they can direct more easily and that can do things in a repeatable loop.
And so these tasks that were previously set aside and often sent to interns or, you know, other junior professionals are being just taken care of by agents in many cases.
It's some of the first work that gets picked up by good AI loops.
And I think as an industry, and frankly, as a world, we have to be thinking about How do we stop there being this gulf between the people who are far enough along in their career and can leverage AI models to 10x themselves, as we put it, versus those who are picking things up and having to build expertise in their skills and in their industries in order to have the leverage to execute these models?
And so, Jonathan, I'm curious from your perspective, as you think about where DataCamp plays a role there and more broadly, how we should be thinking about enabling juniors to enter industries like software engineering, etc.
Yeah, I think this is a really important observation and a really important question.
At the highest level, I would say it's the reason we exist.
We're going through this really fast transition in skills.
And the reason for existence for the whole company is we want to help people who are long humans.
We want to help the humans kind of take advantage of all the new technologies that come about.
And things have never changed as fast as they have today.
If you look at the market as a whole, in the last six to nine months, two things are true in parallel.
So on the one hand, you have this gap for juniors, which is very real.
On the other hand, if you look at AI engineering roles, data engineering roles, they are growing faster than anything we've seen before.
So there's actually an acceleration of certain types of roles where people have AI skills.
But that is true, even though it's harder for junior people to...
to get a new job.
And my personal opinion is, I think it's a big mistake and it's a kind of change management challenge for a lot of larger companies.
You're absolutely right.
Historically, you would hire juniors, you would give them some of the grunt work and that's how they would kind of understand the business and learn the business.
And some of those tasks are also the tasks that are the easiest to kind of automate with AI.
So you have a temporary, gap there.
But I think the right solution for organizations is to recognize that it's the young people that will adapt AI the fastest, that will understand how to leverage AI better, who are truly AI native.
And hiring those people, but just having them skip a few steps will ultimately yield way better results.
Even though you have a short-term effect where some organizations might be tempted to say, hey, we don't need a lot of juniors.
I believe and I hope that's a temporary kind of imbalance in the markets that will hopefully go away very soon.
Any frontier model can query your data.
But can you trust its answers?
This episode's sponsors, Wisdom AI, put that to the test in their latest benchmark study.
Alone, Claude Code topped out at 73% accuracy.
Wisdom AI closed the gap.
delivering accurate answers 3x faster at a third of the cost.
I sat down with Kapil Chhabra, Wisdom AI's co-founder and CPO for episode 75, and he explained the analytics architecture that makes these kind of results possible.
Check it out for a full deep dive, plus get the blueprint for trusted AI analytics at wisdom.ai slash bedspark so you can deliver accurate answers on Claude.
chat to PT or wherever your business gets their insights.
Download your copy at wisdom.ai benchmark.
So as we hopefully solve this problem and Datacamp I know is eager to be part of that solution, this still creates major challenges for the industry more broadly.
And As you work through what the next stage of the tutor needs to look like to help solve this challenge, Yusuf, what are the changes you might make around architecture or model decisions?
What are you bringing in-house on the eval side versus saying, oh, no, we're certainly letting an external team handle that.
I'd love to understand your thinking around architecture and what the future of the platform looks like.
Let me start on the Eva's point, because this is absolutely critical.
A lot of teaching or tutoring, even with human teachers, is a very hard situation to be in as an evaluator.
Like, how can I say that this is actually working, that tutors are performing their duties in the ways that are expected?
So we took this approach, which I think is, is essential for anything we build to work and we broke it into like two problems.
The first question is what should a great tutor do and this is what guides us into how to design these behaviors and then the second question is is our tutor living up to those behaviors and that's where the eval comes in.
So everything we discussed earlier whether learners are actually benefiting from that turning and proxies for that when it comes to like engagement, a willingness to pay a completion of learning material.
That always on the first question which is what are the right behaviors that a good tutor should exhibit and then the second question is now that we know them is our tutor exhibiting now.
So this is another example of a behavior so we looked at one earlier that was about our learner submits their work for feedback.
This is when learners ask a question.
So this is one of the most common behaviors.
So a good teacher handles this moment really, really well.
It's a moment where learners throw in curiosity, they're speaking up.
And if that's not handled well, that could really hurt these learners, not just for this lesson, but their whole attitude towards learning could be impacted.
Once the learner asks a question, the teacher would try to put it in one of four classes.
So the first one, this is a question about what you're teaching right now.
This is, they are trying to understand it better.
They're confused about something.
They want clarification.
If that is the case, then it will answer it in full and it will check that it landed and it will not move on to the next point until it makes sure that this is clear.
The learner is essentially helping the teacher do his job by telling it, I don't get this part.
So it has to kind of cater to that fully.
And the second is the learner is curious and they're asking a question about something that is coming soon.
It's one of the most common situations you have with a bright learner.
They're slightly ahead and they're asking questions about something that's coming soon.
Here we give them a short, concise answer.
We appreciate the curiosity, give a short, concise answer, and then reassure them that this is coming at this moment.
Usually that's sufficient.
We don't shut them down, but we also don't derail the flow of the lesson because there are usually dependencies that need to be addressed.
And third, it's beyond the course.
It's a very good question, but it's beyond.
Again, a lot of focus on appreciating the curiosity and then pointing them to where whether it's in our course catalog or elsewhere, where they can go and fulfill that curiosity.
And finally, something that's totally off-topic.
People sometimes just do it for fun, or they ask something that has nothing to do with learning.
And here the tutor would not engage.
So it would say, yeah, but just fake a laugh and then say, good, let's...
stay on track and continue learning or it will say hey if you'd like to take a break and then you know when you're back.
So that's kind of a behavior.
Now how do we ensure that the tutor is actually doing this in those situations?
This is where the evaluation system comes in.
All right so the every behavior is pinned by a set of expectations.
So the behavior we just saw it kind of gives us a set of four expectations.
So answers fully when on topic.
checks that it landed.
So we expect it to do those things.
So we can say that it actually exhibits this behavior.
It stays brief on our scope or doesn't engage in off topic.
So for every behavior we connect with a set of expectations.
It can be anywhere from like you know three to you know tens.
And then each of those expectations we kind of evaluate it through a set of tests.
Now a test is an actual lesson that has unfolded up to this point.
So a real lesson where you have this lesson started, the teaching is going on, and then we get to a point where the learner asks the question and then we see how the tutor responds that's being evaluated.
So for each expectation there is a set anywhere from you know three to you know low tens of what we call casts.
That s is this whole thing, like it's a lesson from start to that very point.
I just check that it produces the next response correctly.
And we try to assess, what we do is slightly different from this simplified version.
So there are two questions, can it do this and can do this reliably?
So if we give it five situations So each test we run it multiple times.
So if we have 10 tests for one expectation, they can be run each five times.
So and then if it gets them consistently right, then you know, this is, the tutor can reliably exhibit this behavior.
If it gets them sometimes yes, sometimes no, then that means it's not as reliable and we have to iterate on it.
So that's at high level how we answer the question.
If we assume that these are indeed the right behaviors, is it actually exhibiting those behaviors?
And in a way that we can run, this can take several minutes to go through the whole evaluation run.
But then we get something that is close to a rigorous answer to that question.
In the past, we'd have to have a bunch of lessons and then we'd have the team go and take them.
So just to get a maybe answer.
it spent you know four to six hours and then the end maybe it just got them right this time.
I have no idea if it would get them right every other time and we haven't tested every aspect of every behavior.
So I think with this setup we can actually say that we're not sure if we are covering all the right behaviors within our tutor but the ones we do cover are actually working in the way we expect.
Yusuf, thank you for walking our audience through this approach.
I think it is instructive.
As we close out here, Jonathan, Yusuf, I'd love to get some thoughts from both of you, but Jonathan, why don't you start on where you see Datacamp going and the role that you look to play in the future and perhaps just maybe give our listeners, hey, we want to check out Datacamp, where should they go?
If anyone wants to learn about anything related to data and AI, go to datacamp.com.
you'll see kind of the courses that have the AI tutor.
And Conor, to link back to what you said earlier, I think it's a scary time for a lot of people where things are changing extremely fast.
Expectations for jobs are changing extremely fast, faster than ever before, at least in my lifetime.
And so we want to be a force that helps people kind of through these transitions, focusing on...
the essential skills.
And our vision is that we'll do that by building the best AI tutor and then building all the systems that enable that tutor to teach every relevant professional topic.
That's kind of what we're working towards and that's the role we want to play.
Lava, any closing thoughts from you, Yusuf?
If I were to add something to this same point, we spend a lot of time thinking about that.
And we see one in that the nature of the substance itself, like what we teach, is actually changing.
And every new course that we're creating now, we ask this question, what would someone doing this in 2027 need to know?
And that's usually quite different from, you know, what we learned when we did, you know, data science or machine learning or back when we learned 10, 15 years ago.
So it's not enough to create a course for a tool or a skill.
It's also the very nature of using those tools and skills has changed.
I try to see what's the balance point between giving learners enough of the underlying, like really how it works, like how kids learn to do long division by hand to kind of get a theory for that.
So how much of that should still remain in the course, even though it's not done at all in real life, you would not do it.
You would have the model do it for you.
and how much of the course should be about the meta skills, about how I work with models and kind of manage the development process or the execution.
And I think it's still an open question.
And I think it's a responsibility for educators everywhere to teach what we learned, what we've been teaching for a very long time.
I think it's hard to kind of try to reimagine.
What should people learn now that actually sets them up for success in this world?
And we try to do that as much as we can with the courses we are creating.
Amazing.
Jonathan, Yusuf, thank you so much for joining me today.
And for everyone who enjoyed this conversation, make sure you have hit that subscribe button on your platform of choice and maybe drop a comment.
Just let us know what you thought and let us know what you'd want to learn on Datacamp or what you learned from this conversation.
But yeah, Jonathan, Yusuf, great to see you guys and thanks so much.
Yeah, thanks so much, Connor.
Thank you for having us.