# Optimizing LLM Inference with LLMD on Kubernetes

**Podcast:** The InfoQ Podcast
**Published:** 2026-11-09

## Transcript

How much authority should an AI agent have?
At QCon AI New York, learn how senior practitioners set permissions and evaluate agents in production.
Use their experience to decide whether your own agents meet boundaries and checks before release.
Join us December 15th and 16th, 2026 at the Western Jersey City, Newport.
Explore the program at newyork.qcon.ai.
Hello everybody, I'm Olympi Popon, Info Curator, and I have in front of me today, Eduardo Vache, one of the two Italians that usually talk about boxes inside of boxes.
The other one is Andrea Perufo.
Eduardo worked for a long period of time in the WebAssembly ecosystem.
He did some spotting around other types of EMs, and today he's back at Red Hat.
He's trying to make things easier in terms of Kubernetes and inference.
Is that correct?
Yeah, I mean, on my team.
It's a few people working on this, and I'm just one of them.
But fair description.
We're working on distributed inference on Kubernetes specifically, and that's called LLMD.
And it's part of the VLLM family of projects.
So yeah, super exciting to be back at Red Hat and work on, you know, the bleeding edge.
Yeah, that sounds nice.
But before we go there, because that's the gist of your current work, tell us what happens currently in the WebAssembly space.
And I'm asking you because besides working on the Java side of the WebAssembly, you had exposure to Wazero, if I remember correctly.
That's right.
maybe just have like a short refresh.
I know that the WebAssembly space is moving fast.
So maybe if you can just provide a short summary, where it is, what changed and what's to be expected in the upcoming period.
Right.
So for the benefit of our audience, I don't know how much all of them are familiar with it, but you will know that Wasm was born for web browsers, but more recently, and by recently now, it means a few years.
People were also interested in running it outside the browser and for the backend specifically.
So running software on Wasm.
And in that sense, it could be seen as a sort of a similar thing to the JVM in spirit.
And that's why I guess some of us that were working on the JVM were interested in working on Wasm as well.
So Wasm has changed.
fast in some senses.
It evolved quickly in some senses.
It evolved just slowly in others.
So there's been a lot of discussion around the component model, which is this new modular system that's built around WebAssembly.
That is probably where a lot of the development happens these days, I'd say.
But I'd say that the space for WebAssembly on the backend has changed a lot in the last few years.
I think that all of our development landscape has changed.
So things have shifted a little bit.
I think it's very meaningful and relevant today, even more possibly than it was when there was much more hype because it has a lot to do with sandboxing.
And as you know, sandboxing is all the rage these days.
One good scoring point for WebAssembly was earlier this year, I was talking to the local first community.
And what was interesting was that they were mentioning that one of the libraries they built, the name is eluding me now, but what they said is that rather than just thinking about providing implementation in different programming languages, they built it in Rust.
And that meant that they can just compile it.
Right.
To WebAssembly and then use it via WebAssembly to different programming languages.
And that was the nice part from their side.
And they were quite, quite happy with the result.
And I think that's where the power of WebAssembly comes.
Or I think the previous year, I spoke to a couple of guys that used to work on infrastructure and they spoke about Firecracker.
And what was nice is that they said that most often than not, the harder that you have in the server racks is idle.
What they did on top of those idle space, they just had very small WebAssembly machines.
Well, containers, I think it's a better wording for that.
That was doing very small things that were quite useful for other people.
So I feel that there are all those different scenarios where people are using WebAssembly.
Yeah, I think one interesting aspect that connects to what you were saying right now is even migrating workloads somehow, like running stuff in the browser.
But then you also have these WebAssembly runtimes that can run even on devices or in general outside the browser.
And you have the same workload that runs on your browser.
And we have demos of things that are migrated, even live migrated.
from the browser to an end-user device without breakage.
That's really fascinating.
And I think these days when we talk about agentic workloads, that's an interesting space.
That's where I think WASM is still relevant.
I mean, WASM is still relevant, especially for...
browser-related work and many other.
But these scenarios, you know, for the world that's changing a lot these days for developers, those are the use cases where really could shine.
Yeah, as you mentioned, isolation is important and is getting more and more important these days.
So I think that's quite useful.
What is bugging me for at least one week now, and probably I'll try it the following week.
Let's see what you're going to say to that.
I'm just thinking to run a small language model inside the WebAssembly in a browser.
Is that feasible?
Is it not feasible?
Yeah, I mean, it is feasible.
There are examples of that.
The thing is, I've been on the receiving end of this question a few times now.
So it depends is the short answer.
WebAssembly is meant to run on the CPU first.
And as we know, We can do inference on the CPU, but generally it works much better.
And for the way LLMs, modern LLMs, I mean, the modern architectures for LLMs are designed, they are meant to be run with highly parallel architectures.
And the CPU is parallel, but not as parallel as a GPU, right?
So long story short, smaller models, which basically, make sense to run in a browser, for instance, do make sense to run in WebAssembly, much bigger workloads.
Well, those in general, we probably want to run on a GPU anyway.
However...
There are experiments that I'm aware of to translate WebAssembly into GPU kernels.
So, yeah, possibly.
But the short version of it is for all those workloads that are meaningful to run on a CPU, so relatively small language models.
WASM can be used, and there are many versions of these workloads that exist.
I think that hugging face.
For some of the smallest models, we'll run them on your browser, I think.
And there are a few examples.
On the Wasam.io conference, there were quite a few talks of people showing this, running these workloads in the browser, especially to do inference.
So yeah, it does work.
It depends on your workload.
Yeah, yeah.
Well, I'm going to say it.
Anyway, it's simple.
It's more about just trying and seeing how things evolved.
And fingers crossed that something will happen and not just smoke.
But in order to work and consolidate the point that I stated about an expert into putting boxes into boxes, we all had a post, I think last year or the previous year, about Tornado VM.
I know that you just had a small exposure to it, so maybe just have a short summary.
What's Tornado VM and when should we use it?
Yeah, so Tornado VM is, we could say, a flavor of the JVM or an add-on, a plug-in to the JVM.
Depends on how you see it.
Tornado VM allows you to run Java code, essentially, on the GPU.
This is the bottom line, the selling point, you know, the very short version of it.
It's a little bit...
more nuanced than that.
But yeah, basically you can, speaking of boxes inside boxes, yes, it's called the VM because it allows you to translate on the fly your Java code and compile it on the fly into GPU-specific kernels.
So PTX for NVIDIA, Spearbee, whatever.
And it handles for you this translation.
So you can basically write Java code and then it will translate for GPU backend.
Now, the code that you write there, it is Java code, but you have to write the code in a specific way.
You might have to use some libraries that Tornado VM provides you.
You cannot run basically any code.
You won't be able to run Spring on Tornado VM.
That's the bottom line.
I mean, yes, but it will run in Java mode.
You won't run Spring on the GPU.
That's the point.
But yeah, you can write GPU kernels.
using Java.
And that's the beauty of it.
And the blog post you were referring to, I think it was last year's.
I'm walking through the reader.
One of the examples in the code, I think there is an example of writing a simple kernel and running on the GPU.
That's one thing that we show in the blog.
But then we run GPU Llama 3.
which is an implementation of the LAMA 3 model that runs on GPU.
And this implementation is completely written in Java.
So that's fascinating.
The blog post walks you through how this works.
In the latest DevOps Belgium, if people want to catch up a little bit, they have demonstrated, I think there was definitely a talk about Tornado VM.
There were talks from the Growl VM team.
of the similar projects to run LAMA and other language models using GraalVM that will do the acceleration for you.
And then there was also a former GraalVM team member that spun off his own startup.
I think his name is Alfonso Alfonso.
I remember it because his nickname is Alfonso Squared.
And he started his own startup, branching off some work that he did originally in Growl VM, again, to run code, Java code, I think, on the GPU and LLM.
So, yeah, fascinating stuff.
And so bottom line, the reason why I started to look into this is fun of boxes and sandboxes, fun of languages and compilers.
So it felt like a natural evolution of the things that I did in the past, including compilation to WebAssembly to learn.
about compilation to the GPU.
That sounds quite interesting and definitely makes sense these days.
For instance, I saw that a lot of the big companies are just using Java for inference and that makes perfect sense to just have this close in the loop.
Today, you are part of the team at Red Hat, back again at Red Hat, that is working on LLMD.
And in my understanding, what LLMD is, Taking a step back here, VLLM is making sure that everything runs perfectly on one box.
LLMD is trying to make sure that everything is efficient if you have a Kubernetes cluster.
Is that the correct affirmation or where did I understand wrong?
No, no, it is correct on a high level.
More specifically, so VLLM is our inference engine.
So you can think of it, if you will, as the JVM of language models.
So it has support for several language models.
There's an interesting aside here.
It is a platform to run LLMs.
The interesting aside here is you would expect that this is a general sort of VM and the LLMs are sort of software that you load on top of it.
But unfortunately, every LLM architecture is slightly different.
So the LLM has to basically include the implementation of every LLM that gets released.
So when people release LLMs, they do not release the software to run that, they release the weights.
So all of these inference engines, such as VLLM, what they do is implementing basically the software, the kernels, the components that actually load the weights and then actually do the inference.
So VLLM is our platform to run the core.
And effectively, VLLM is able to serve.
It's a server.
I mean, it's an inference engine.
It's also a server that exposes endpoints that you can call to do inference.
And it does so efficiently.
It was originally VLLM, I think the V in VLLM originally stays for virtual, as in virtual memory allocation, because it was the first engine to implement paged attention, basically a mechanism inspired by good old systems programming to do paging of memory, memory pages, but for memory that gets allocated on GPU.
This is very, you know, in a very high level.
level way.
So BLM is able to do high-throughput inference.
So you can do several requests at once, and BLM will pack them and try to serve them together and do inference in one pass for several requests at one time.
This is called continuous batching.
But this gets you to a point.
Essentially, at some point, you have your GPUs on your machine.
and your inference engine, and that's it.
You will not be able to scale further than the resources that you have on that machine.
So how do you scale further?
Well, you can scale by having more VLLM instances, right?
So you have several VLLM instances, and you want to serve your inference requests using these few VLLM instances that you have.
But now you have another problem.
How do you route requests across your VLL instances?
So if you are on Kubernetes, what you will do, and most people are on Kubernetes these days, the standard for platforms for serving services in an efficient way.
So you can scale your VLL instances and have several VLL instances that serve the possible same request.
I mean, that gets selected.
by the scheduler, by the VLLM scheduler.
If you do it the traditional way and let the Kubernetes normal router to pick across the VLLM instances that you have replicated, you won't get necessarily good performance.
The reason is inference does not behave like a traditional service, a traditional HTTP service.
Basically, if you send a request, to one specific VLLM instance, especially if you are doing conversation, if you're doing a chat, and if you're doing agentic coding, it's still a chat, basically.
There's a lot of exchanges back and forth between you and your server, between your agent and your server.
You want to stick for as much time as possible to the same VLLM instance.
This might be...
counter-intuitive in a way, because for microservices, we know we don't want to hammer the same service on and on because otherwise it will get overloaded, right?
We want to do sort of round robbing or random robbing across the replicas.
But the thing is, microservices or service in general, these are usually in a way stateless.
The state is usually stored somewhere else and it's not part of your request.
I mean, the request can be served independently by any instance.
Unfortunately, in order for LLMs to do their inference in an efficient way, you want to stick for as much time as possible to the same instance.
But at the same time, you cannot stick forever to the same instance because at some point you will reach some limit.
You are not the only user of that BLM instance.
You want to serve and distribute the load across your BLM instances.
So there's a balance that you have to strike between staying with the same instance and benefit of the caches, the KV caches, the prefix caches that are in place, the systems that allow you to do inference in an efficient way and lower costs.
But at the same time, you don't want to overload the same instance too much.
So you also have to keep this into account.
This is not the job for a traditional Kubernetes schedule.
And that's where LLMD comes into place.
That's what LLMD does.
LLMD is not just a router or a scheduler.
It's actually a family of projects, and they are all meant to give you a set of tools to do distributed inference so that you can leverage all of the GPUs that you have and use it at their maximum, as much as possible, so you get...
the best value, the best bang for your bucks, because this hardware is expensive, whether you loan it or you own it.
So you want to use it 100% as much as possible.
That's why we provide scheduler.
We have something that we're going to release.
I mean, in the open source ecosystem, it's already been released.
We're going to release also as part of product of Red Hat.
This is called batch inference.
Batch inference is using your cluster even when it's under load.
So you can schedule requests to do batch processing in the background when your cluster is underused.
Because basically this hardware, this hardware is so expensive that you want to keep it at 100% of the time.
So even when there is no live inference going on, you can schedule batch processing to run when you're sleeping, when there's less work to be done live.
So that's what LLMD does.
It uses your hardware as much as possible so that the money you spend gets you the best value.
And you do it on your own hardware so you can do inference without leaking data to a third party, which is, I mean, important these days.
So what you're saying is that we will be able to use our previous Kubernetes clusters.
So what we still have from the cloud native era, because everything was native these days, and just make sure that we can rebuild it to build our privacy also for the AI native era.
Yeah, Anthony, OLD allows you to make cloud native inference and evolve your infrastructure in such a way that you can do AI on-premises in a similar way as you did cloud native development before AI.
Well, that was always an issue, but now it feels that we are discussing in the cloud native most of the times we spoke about stateless.
Whenever you call something, you didn't always ring on the same doorbell because that was one issue.
Another issue was in the Kubernetes space, it was the part with IO.
Well, most particularly persistent data like database and stuff like that.
Whenever you spoke to infrastructure guys, they told you, well, we can have everything in Kubernetes except for the databases because then those things can become more complex.
But let's take them on by on.
How do you bring in a stateless ecosystem like Kubernetes worked a lot in Bing something that theoretically should be stateful?
Because if I have a conversation, I would like to continue my conversation about, I don't know, whatever.
stupid thing I speak to Gemini or Cloud or OpenGPT or whatever stuff.
I don't want to have my responses coming for what Eduardo was speaking to his LLM.
Yeah, it's a complex problem for sure.
As I said, one part of the story is that, well, I mean, if people that are hearing the audience know how LLM generally work, essentially the Stateful nature of the yellow lamb in a way is fake in that they are really stateless.
The way they do inference is that we send them the entire history of a conversation every single time.
So maybe I jumped at the conclusion.
Yeah, to the conclusion earlier, and I said a lot about prefix caches and KV caches, but yeah.
So essentially, the way LLM works is that essentially you always send the entirety of the conversation to the LLM.
And what really happens is the LLM completes the conversation.
I don't know if people recall how OpenAI and the GPT Lab playground worked a few years ago.
Basically, you only wrote, it was the completion API.
You wrote a sentence and then the LLM wrote the continuation of that.
And the cool thing was that someone discovered if you wrote that prompt as a conversation, as a dialogue with quotes, user says, da-da-da, computer says, the LLM will continue the conversation.
And that's how ChatGPT basically was born.
Like people notice this and notice this pattern.
Now, LLMs are being trained with this pattern, but that's how basically originally it worked.
So if you think of it this way, every conversation really is a very long prompt to the continuation of that conversation.
The way LLM works, you send the history of the conversation and then they continue with the next answer.
And then you get back.
the answer, and then you continue with your answer, and so on and so forth, so on and so forth.
If you are using LLMs, you will know that pricing for these things, depending if you're using token-based spend or a subscription, well, if you're using a subscription, that doesn't really matter.
But if you're using token-based pricing, you will know that there's a difference between cached responses and responses.
The reason is If you don't do a cached response, then you are sending the entire history of your conversation to the LLM.
The LLM does not have that conversation in cache, so it will have to do inference on the entire prompt.
And that's why that will be particularly expensive.
But if you've been going back and forth for a while without long pauses and the cache on the server has not been evicted, then you've been reusing stored cache state to continue the conversation.
That does not mean necessarily that you haven't sent the entire history to the LLM.
It just means that even though you are sending the entire history, the history is recognized as part of a prefix of a cache.
And so the LLM can compute only the continuation instead of the entire prompt.
So we have these mechanisms for caching in place so that basically LLMs, if the cache exists, can only infer the last few tokens that continue the conversation instead of have to recompute the entire state from the prompt.
LLMD does that for you.
So it maintains this cache both in the form of a KB cache with one going to the details, how that works, and a prefix cache.
And both these caching mechanisms allow you to persist this state across the exchanges in your conversation, persist as in volatile memory, because these are caches.
Usually they're not persistent for a long time.
And that's the reason also why you want to stick usually to the same BLM instance, even though LND also gives you ways to move around across instances.
It's a lot.
We cannot explain how all that works.
But in general, LLMD allows you to do this, allows you to have an efficient use and even for stateful conversation of your hardware.
Okay.
So LLMD, it's also donated, if I remember correctly, to the Ciencia Foundation, right?
Yeah.
It recently reached 1.0 a few days ago.
Yeah.
Okay.
Congratulations.
So we are right in time.
In what state is it?
Is it production ready?
Should we wait for it?
Oh, no, it's definitely production ready.
There's a number of companies that are already using it.
At Red Hat, we're actually partners with NVIDIA for a bunch of things.
We have a lot of success stories already, people using it in production.
And yeah, you should definitely take a look at it.
In fact, people are, sometimes I go to conferences and people, I realize people are already using it.
Or maybe they're using VLLM on their infrastructure and they're not aware of LLMD.
So VLLM allows you to do inference up to a degree in a very high performance, high quality way.
And the LLMD and VLLM projects are sister projects in a way.
I mean, there's a lot of exchanges across the stack.
So we are part of the same team in a way.
But yeah, LMD gives you that edge.
For instance, VLLM does not do automatic flow control.
Even if you're only running a one VLLM instance, LMD gives you control over quality of service and prioritization of HTTP requests.
So that is something that VLLM, at least until a few days ago, I don't know if things have changed recently, could not do.
You can get benefit from LLMD even if you're not necessarily running in a very highly distributed situation.
So what does that mean?
Even if you don't have a cluster, use LLMD?
Even if you don't have a lot of GPUs, even if you don't have to distribute the load across different GPUs, LLMD might still be useful.
And yes, you can also run it bare metal.
So you might want to evaluate LLMD even if you're not using Kubernetes.
I mean, Kubernetes is the first target, but it is supported also on bare metal.
Okay.
So it's focusing on inference, even if you don't have already existing infrastructure.
So it just tries to provide the best usage of your resources, even if you have or you don't have an infrastructure already existing.
It seems that at the point when we are recording this, it's end of October and it seems that LLMD started at the end of October in 2024.
So it's a good moment to have the first major version out in this period.
So congratulations.
Yeah.
But staying in the Kubernetes space, one of the reasons why people prefer to use things that...
They are already familiar.
Kubernetes is not that old of a solution comparing to other types of solutions that were out there, but it's still one of the most mature in the cloud-native space.
And one of the things that are very important these days, or they always were important, but now they are more than ever, it's observability.
What should we expect from that perspective in terms of observability when we're discussing about LLMD?
I mean, as you mentioned, everything related to LLMDs.
more complex than everything else.
How can we make that complexity easier to digest through the observability solution already existing?
Yeah, so, I mean, it depends.
It's a very wide question.
So the way LND actually works, it leverages a lot of signals that are metrics-based.
So there's a lot of insights that are being exposed by BLLM.
Metrics and data that...
explains essentially what's going on inside the inference engine.
And LLMD uses this information to actually do what I explained earlier, that is this informed routing across instances.
What this means is essentially when you use LLMD and VLLM, you don't just have a system to do efficient inference, you also are getting for free, if you will.
a rich system to get insight on what's going on, not just on your cluster, but in your inference engines.
You can see how many tokens are being produced, how many failures you're getting, what's the throughput.
You can get alert on a situation where you might want to intervene.
For instance, you have control over the saturation of your cluster, which means saturation means essentially you have so many requests that you are struggling to serve them.
And then at that point, you can add more machines, you can throttle the incoming requests.
So you have a lot of knobs that you can work on.
The observability story for LMZ is pretty interesting as well.
And you have all the tools to get insights of what's going on in your cluster from a GPU and inference perspective as well.
Well, you already mentioned that it was complicated to get it right.
But which thing was the hardest?
That's an interesting question.
Yes.
Everything.
Well, one thing that is interesting in this ecosystem is that everything is very new.
I would say one of the things that are challenging, maybe not the hardest, is that, well, NVIDIA is probably the biggest, most important vendor.
But there's also AMD, there's Intel, there's Google's TPUs, and there are more.
There are also a lot of startups that are building their own hardware.
And while for the PC, we benefit for a lot of time now of a standardized set of hardware components with a very well-defined instruction set, for these accelerators and GPUs, The languages are very different.
So one of the challenging parts for VLLM itself and inference in general is that, and the reason why we need a framework on top of that, is that you can't write the kernel for a GPU computation once and target every single accelerator there is because everything is different.
It's like having Intel, but also ARM, but also RISC-V, but also PowerPC, and you have to use all of them.
depending on your vendors and you have to support all of them.
I mean, in our case, you want to support as many as possible.
So that's probably one of the most challenging things that happen in this case.
The other thing that is also possibly challenging from a hardware perspective is that there's a lot of data that has to be transferred.
Like GPUs, LLMs are large language models and we hear about trillions of weights of parameters.
These translates in gigabytes, terabytes, petabytes of data.
And these data have to be moved across machines.
You don't just have expensive GPUs, but you also have to have expensive connection between these GPUs.
We're talking about RDMA.
So you have to connect this hardware with very expensive type of connection that allow you to have very fast data transfer.
So I would say, In general, from an artwork perspective, these are things that are very challenging.
At the same time, it makes this space very fascinating and interesting.
Yeah, that's for sure.
But one, you did mention that the beach of possibilities is quite wide when we were discussing about GPUs, which were intended for something else.
But now people, like usual, they adapted and they use it in a different perspective.
Yeah.
But now we can see that there are companies that started building their own harder.
We are discussing about TPUs.
Right.
How will that change the game?
Yeah, that's an interesting question.
I don't know.
Hopefully, it might make things more affordable.
I mean, competition is always good.
Like the one side is that this makes things challenging for people that are building infrastructure and software on top of these things.
It makes things challenging for people that want to source this hardware because it's still very expensive.
But at the same time, if there is more competition, hopefully prices eventually will go down.
I think one thing that is very important, and I think this is also the message that we want to...
bring across with LLMD, with BLM, we're getting to a point where, first of all, you don't really need extremely expensive hardware to do inference to a certain degree of quality.
Obviously, frontier models will always be more capable than older models.
Open models are getting better and better, even relatively small models that can run on consumer hardware.
Like I have a relatively beefy Mac myself with 48 gigabytes of RAM.
This is a company Mac, so I didn't pay for it.
And it was probably bought before the price rises.
So it was a good time.
But it's a relatively small amount of shared memory because it's shared across CPU and GPU.
That's the point.
27 billion, I think, 27 billion, which is not huge, but it's pretty capable.
And people have talked recently about QN 3.8 27B.
This does not run very well on my laptop, but it runs on laptops still.
And it runs on my laptop, even though it's not super fast.
So if you get a server that is only slightly bigger than my laptop, you will be able to serve a lot of requests in a very efficient way.
And this model has been compared to Opus 4.6.
And Opus 4.6 is only one year old.
And we were super excited and impressed by Opus 4.6 last year.
So let's not get to...
carried away by all the announcements we read on social media about how all of these models are super capable and the old models are no longer useful.
The old models are still useful.
And they're even more useful if you can run them on your hardware, because this is very important.
You don't have to share your data with a company that you don't know you can really trust after all.
And this is, I think, the message we want to bring across.
You can own your own AI.
You can own your own token inference on hardware that now it's relatively, relatively is the word here.
Yeah, affordable.
And this is especially true for companies.
I'm not talking about consumer.
This is true for consumer to some extent, but it's very true for companies.
They can source the sort of hardware.
They can do inference for these models.
They're not as big as the frontier, but they're definitely.
quite something and it can solve a lot of interesting problems already.
Yeah, what I think it's important for people to understand is that do we actually need frontier models for everything?
Exactly.
Because in the end, as you mentioned, it's becoming tangible that you can run them on even laptops.
Yeah.
I was mentioning, it's something quite basic.
It's ordering, I don't know, something.
And then you don't actually need anything else but details in terms of human understanding and then a very small model it's more than enough but it's very niche and then that's that's the thing is that yes we do need a perspective where we do that stuff and we need i don't know the argons the opuses and all the other frontier models out there but now a lot of people are doing experimentation and at the point where you're getting into the production space You have to do the refining.
You have to do the efficiency part and then definitely get to the right, on the right spot.
And I saw something similar, even on the edge side, where they had very tailored models that were just being focused on a very thin, narrow use case.
And that's it.
But now we are just, a lot of us are exploring and we are measuring stuff that actually don't matter.
For sure.
I think another interesting aspect, this is another project that we are incubating in LLMD.
It's called LLMD Semantic Classifier.
It deals with routing, not just efficiently, but to the right model for some definition of right.
So right might be cost-based or it might be topic-based.
And so this system, there's another sister project that's called the VLLM Semantic Router, if you're interested.
LLM Semantic Classifier is sort of a sibling, if you will, with different targets.
But the goal is the same.
You have a prompt, you have a conversation going, and you want to route this conversation to the right model, which might mean right-sized.
And that's exactly what you were saying.
So it doesn't really matter.
Sometimes you have a very large model that is able to answer every possible question.
It might be useful in many situations, of course, but you have also to bring to the table the cost efficiency.
So you might have specialized models that are trained or fine-tuned on specific domains.
And I think that's where we should go, where the next thing should be.
Because that really brings LLM inference, AI in general, agentic stuff to the people.
It allows you to run this stuff on your hardware with things that are attainable, goals that are attainable, not just super expensive hardware that nobody can afford.
I think we are at that, let's say, teenage years of what LLMs and AI will mean for the future.
Because everybody is priding themselves on how much horsepower our models have.
But actually, we are stuck in traffic hours and it doesn't actually matter if it's there or not.
Yeah, teenage years is a good metaphor, I'd say.
Maybe just to close up with what's on your roadmap.
So version one is out.
What's in the next period?
So upcoming months, upcoming years, what's the target for LLMD?
Yeah, I mentioned a few already, actually.
So LL&D itself is able to run, you know, because of this hardware, you have to use it efficiently.
It's able to run very large models, mix your expert models by sharing them across machines.
But things that we are experimenting and we are getting more and more refined are this batch inference concept.
So you can leverage your hardware more efficiently and use those dead moments in your traffic.
to serve and do inference, even scheduled inference without the live interaction so you can 100% reuse that hardware that you have.
And the other is obviously, as I just mentioned, semantic classification so that you can not just use the right machines, but also use the right model at the right time.
For instance, let's say you have a very big model that is expensive to run.
And smaller models that are more efficient to run and you can run on hardware that is much cheaper.
Depending on the conversation that you're having, on the prompt that you're sending, you might want to use the larger model in some situations or the smaller model in some situations.
So these are the things that we are working on currently.
There are more and I cannot even list all of them, but it's an exciting space and I think you're going to see a lot of interesting stuff in the next coming months.
Okay.
Thank you for your time.
Thank you.