# Liquid AI: Edge-First LLMs and Hardware Co-Design

**Podcast:** alphalist.CTO Podcast - For CTOs and Technical Leaders
**Published:** 2026-11-12

## Transcript

Hello friends, this is the Alphalist podcast.
I am your host, Tobi.
The goal of the Alphalist podcast is to empower CTOs with the info and insight they need to make the best decisions for their company.
We do this by hosting top thought leaders and picking their brains for insights into technical leadership and tech trends.
If you believe in the power of accumulated knowledge to accelerate growth, make sure to subscribe to this podcast.
Plus, if you're an experienced CTO, you will love the discussion happening in our Slack space where over 600 CTOs are sharing insights or visit one of our events.
Just go to alphalist.com to apply.
Welcome to the Alphalist podcast.
I'm your host, Tobi.
And my guest today is Matthias Lechner.
Matthias is an Austrian, did his PhD in Vienna on the nervous system of a worm with 203 neurons.
and turned that into Liquid AI, an MIT spin-off that builds foundation models running on phones, laptops, and cars without the cloud.
AMD led their 250 million round at a 2.3 billion valuation.
So you gave away.
roughly 10% of your shares.
And Mercedes shifts their models in the new CLA this year.
And I think Time just chose you as one of the Time 100 AI List members.
Matthias, welcome.
Did I forget anything?
Anything to add here?
No, I think that's a great injury.
Thank you so much.
It's great to be here.
Okay, thanks for being my guest.
Maybe we start like...
A little earlier, I don't know how old you are, maybe you add that to the answer as well.
Because he looks quite young, right?
When you were a kid, what sparked your interest?
Or was it like a kid age?
At what age was it?
What sparked your interest in computers?
And how does a kid in Austria end up simulating a worm?
Yeah, that's a good question.
um so yeah i'm like uh right now like 33 years old so it's like i may look young but you know hopefully i'm also still young inside but uh you know my the clock shows something else um yeah like uh how i come into uh computer science in general like um computers in general they were super interesting for me as a kid because i think they appear um like magic you know you can interact with something you can interact with something that's not a that's not a person that's not like a with a machine kind of and it was always you know something that's i think close to magic like when i grew up there was this harry potter books and so on and like they had this kind of um you know like uh like pictures on the wall they can talk and so on and like the computer could do the same thing so it was kind of that you know that's very interesting for me like to work with magic and um so that's why i started computer science and uh how i came into the contact with this warm or like this uh nervous system uh in of the c elegans in general was um that in vienna actually there's a neuroscience research institute and they are working actually with this one like it's called c elegans and has 302 neurons and around 8 000 synapses so it's a very very small you know like nervous system like with very few notes and so on and it could do Or these worms can do actually quite remarkable behaviors, like from like a social, they come together when they sleep, you know, they actually remember, you know, certain salt concentration with the presence or the absence of food and so on.
So they can show it like this small number of nerve cells, very remarkable, robust behavior compared to like artificial intelligence back then in 2016 and so on when we started looking into this.
who were like massive networks that actually can be fooled quite easily by just perturbing the input image.
You know, there was this classical example of like the stop sign where you put on a sticker and now the neural network doesn't classify it correctly anymore.
And kind of the remarkable robustness of this one that actually sparked the interest of my PhD advisor or like PhD, but master's advisor.
And also now co-founder Daniela Roos.
who said, hey, this looks very interesting.
Let's investigate this.
Let's look into neuroscience research and try to bring them into machine learning, try to bring them into robotic learning.
And so basically they announced, hey, we have a position for a PhD student, for math students and so on.
And this is how Ramin, my co-founder, and then also me as a math student, got in touch with Radu and Daniela because they were basically trying to look into this robustness.
So they kind of did this intriguing behavior of this one.
I think you're at a certain point, like did a self-driving demo that used instead of 302 neurons, used just 19.
And so the whole industry basically went the other way.
What did the NRC that Silicon Valley didn't?
Or was it like, basically, did you both have that idea at the same time?
Or like, how did that happen?
Yeah, like I think the car is, I think it's generally interesting kind of artifact because like when we started looking into this, we were actually at TU Wien, TU Vienna, TU Wien.
And, you know, they don't have a car.
They only have like small robots and so on.
And like one of the reasons, you know, we like how me and Ramon, you know, got in touch with MIT was basically that, you know, like our work on the robots that we had actually access to in TU Wien.
we were kind of limited, we kind of exhausted the possibilities there.
So that's why like the Bani Alliance said, hey, let's come, please come to MIT.
We want to scale the application to a car.
And this is how we kind of basically then worked on like this scaling, this idea of like this neural network inspired by the C-Elegance network system toward autonomous driving.
And like we basically came up with this network of like 19 neurons.
Like there's like obviously like a bit of...
convolutional layers in front of it that also have like a lot some complexity in there to kind of reduce kind of the high dimensional image into like a small dimensional space but the entire control network kind of the takes care of the steering and so on the robustness this was like 19 neurons and there was as you mentioned at around like that age like 2015-16 there was like the scaling laws kind of emerging um with with dario amuday and so on from from anthropic one of the first papers that actually looked into like um that more systematically like that you actually if you scale the model size the performance of the resulting model becomes better even though if you fix kind of the data set that you have available and so it was actually kind of exactly the opposite direction but i think there's like a a nice gap in a sense of of what was the primary objective i think with the scaling laws i think they're still true and so on like if you scale capabilities and fitting data knowledge specifically You need more parameters, you need more things that actually can capture.
But at the same time, they introduce spurious correlations and so on.
But actually, if you reduce the number of neurons or reduce the size of the parameter of the network, you want to control for that.
So basically, there's this kind of, do you care about the robustness, the controllability aspect, observability aspect?
Or do you care about just pure knowledge in a way that you can capture?
So I think there's these two paradigms going on.
And even to some extent, they're still true to this day.
And the self-driving basically is like less generic and thus basically can live with less parameters and less neurons.
Yeah, exactly.
So if you have a language model, you need to remember, you know, like that, you know, when Einstein was born, kind of that kind of fact.
But in a car, you don't need that, right?
You want it to be very good at like just making sure you stay on the lane, you know?
Even though in our case, there was extreme light perturbation and so on because the sun was very low in the forest.
Actually, that's why we tested the car.
So there was a lot of shadows and so on that cast a normal neural network, a convolutional network that you just train on a couple of hours of data basically fail because of the presence of these shadows and so on.
And so you want actually to be very robust in that sense, but you don't care about the facts of the world in that sense.
And is vision generally enough for such a model to operate?
What's your take on that?
Depends on the task.
I think the autonomous driving, if you just keep on the lane, it's a much simpler task than way more.
So that navigates in an urban environment because there's much more than just staying on the lane.
So it depends.
If you want to just stay on the lane, I think vision is probably enough.
But if you run into more complex scenarios, I think you need probably more or multidimensional sensors as well, like multiple cameras and so on.
And for us, it was also interesting, as I mentioned before, at TU, we had these small robots and so on.
We exhausted them and then we moved to a METI and we scaled it to an autonomous car.
But even there, I would say, because the test track is very small, very limited, and we cannot test on the public roads, right?
Even that, we reached kind of like, okay, what to do next?
kind of point.
Okay.
Okay.
And then you started off a company, you named that after Liquid Neural Networks.
So it warms brain and math.
And then published a model called LHM2, which basically is like a bit of convolutional layers and attention.
Is there any warm left in the equation anywhere?
Anything that still is attached to that one's behavior?
That's a good question.
I get actually asked this a lot.
Yes, it is.
Because one of the things, actually, when you look at the warm neural network, the neurons itself, how the neuroscientists model them, they're actually quite similar to how artificial intelligence, artificial neural networks are modeled in machine learning.
they're like a cell that has like a capacity like there's like some um you know temporal behavior and so on going on there and it sums up kind of the inputs into the cell and so on so it's very i would say very similar to kind of the way the neuroscientists study kind of this elegance at least in this paper that we use or this set of papers have used and in the artificial learning but the big gap is in the synopsis the way you know um in machine learning, the way a synapses model is just a weight connection that just multiplies with the input and then adds to the output.
But the way neuroscientists study synapses or model synapses is way more complex.
There's neurotransmitters, there's different types of neurotransmitters.
They release this chemical into this kind of gap between, like it's called synaptic left, between the post-synaptic and pre-synaptic neuron.
And the amount of neurotransmitters can saturate.
in addition to kind of the saturation of the neuron itself.
So you have like a second kind of saturation effect that you can have there.
And that kind of the way the different ways to model it in a way, but like one way that we know a liquid neural network that is small is kind of a multiplicative gate.
And that gating is still actually present in LSM2 models, you know.
So it seems kind of you model the things that the essence, I would say that's still in LSM2 is like the interaction between two cells, between two units of compute.
needs to be modulated.
And that modulation in a way is still kind of from the original liquid networks, from this Synapse model that is still found kind of in the LFM2 model because it gives more power kind of to express more complex functions like a smaller set of nodes.
And does that form into a mode or what is your, like if I would ask you for a technical mode of the company, what is that today?
Is it connected to this or?
Is it basically, yeah, what is it?
I think in, at least because I'm a researcher, you know, like for me at least, there's no long-term technical mode anywhere, like anything, right?
It's like, it's just innovation.
It's a race, you know?
So that's why like there's a race going on between all of these AI companies.
And how we're trying to, like right now we are staying ahead a bit because we had this LFM2 designed to be very efficient on the bus.
So that's why, for instance, the convolutions.
that we use in the LFM network, this has not been used by any other model.
Like if you go to DeepSeq or OpenAI models or like this, anything that's open source, at least there's nothing that uses convolutions the way we do it.
The reason we chose this is basically it's derived from an architecture search because we just connected basically our cluster, our training cluster with a phone, such that we can actually kind of search for better architectures by like looking into cluster, like training models, candidate models, like whether they fit the data.
and they're connecting it to the phone to actually say, is this model actually running fast on the device?
And it turns out the convolutions are extremely fast because, you know, like a lot of signal processing, a lot of stuff that happens kind of currently, like on a lot of smartphones, is based on, is implemented through convolutions.
So that's why the hardware is already there, which kind of makes sense.
So that's why we have this LFM2 models, like, and they're extremely efficient, like they run on a Raspberry Pi kind of, like imagine a language model that has like...
instruction following capabilities of GPT 3.5 level, like very old, big models on a Raspberry Pi, like at 40 tokens per second.
Like that's that kind of level of...
Sorry, how many tokens per second?
Around 40 tokens per second decode.
40 tokens per second.
Okay, wow.
That's cool.
Yes, it's nice.
And yeah, so that's why for us, the current...
strategy is kind of to build efficient models but as i mentioned before there's no mode the way we think about the mode is actually about silicon partnerships because the thing is this if you want to build the fastest models you know you need to really integrate it at the hardware level like with the with the manufacturer with the silicon provider because like there's so low level you know access kind of to the npus so kind of the hidden units data compute instructions and all of this special you know silicon like the cache lines the cache fetch rate and all of this optimal level stuff that you can only really access if you're working with hardware companies.
So this is why, for instance, our current strategy is to work really closely with this.
So for instance, with Qualcomm, last week, like Ramin, our CEO was on stage with the Qualcomm CEO at the Snapdragon Summit.
So it's like CEO to CEO level communication and partnerships that we have with AMD, with Qualcomm and a lot of other companies that are out there.
But is that exclusive partnerships or what does it mean that you basically get access to their hardware, but like...
what do they get in return?
Exactly.
Basically, they try to sell more hardware, right?
So, in the end, they're trying to sell more hardware.
But they're like, hey, what we get from them is basically resources.
They help us basically optimize our model on their chips.
And also, we get access to unreleased chips.
So, we have with some of these companies, I mentioned, exclusive, or not exclusive, but basically access to the pre...
released hardware.
So we can already start optimizing and testing our models on the hardware that's coming out in half a year or something like that for now.
And at the same time, we also get support from the engineers to say, hey, what's coming and so on.
With other chip manufacturers or silicon providers, we are working basically for the next generation.
So we are designing together with them some instructions and modifications.
This is not public yet, but there's a lot of really nice work.
I'm very interested, at least because you would assume or I would have assumed that these companies move very slow because like the cycle to manufacture, to build the chip, right?
It's very long.
But they are, I would say, quite fast in a lot of aspects.
Quite agile.
Okay.
Yes.
Okay, cool.
And if you optimize your models, then I understood that you're...
Like for you, training is rather the small part, but inference really dominates the lifetime cost and your view on how an optimal model works.
So your North Star KPI is really like inference cost and speed of inference?
Yes, basically speed of inference.
And it's different from a normal open AI or train the model and the optimizer as well for inference.
but for the sake of reducing the costs and being competitive to Chinese models or to Anthropics or to other models and so on.
But for us, it's really to enable it to run it locally.
That's our difference.
We want to run, you want to achieve, let's say, 40 or 100 tokens per second on a certain hardware.
That's our goal.
And where do you see the world heading?
I mean, right now, there are a lot of talks about local inference and edge compute, etc.
Do you think it will stay?
Do you think it's realistic that, I don't know, on this device in a few years, I run most of my workloads?
Or what do you think?
What is your prediction?
I think there will be workloads both in the cloud and for very large models as well as local.
For instance, a coding model.
One thing that I think is very fascinating for me is when we released our models that are small, they run on their phone and so on, people want to do coding agents with them.
It's like, why would you want to do a coding with a 1 billion par meter model?
Please use a bigger one.
Don't use our models.
Use something that's bigger, or at least use our very largest model at least.
Because there's much more capabilities, much more knowledge that is required to do a coding agent well.
So for those kind of tasks, there will be always, I think, need for large-scale cloud models and so on.
but a lot of tasks, a lot of intelligence actually can be packed on a phone.
So really, I think that especially with voice agents, basically when you talk and record stuff, you really can...
I think there's a lot of unlocked potential and specifically also the privacy aspect as well.
So I don't want the phone, my phone or my device to listen and send it to the cloud 24-7.
But I wanted to run 24-7 to do and summarize and take notes of what I'm talking about because it can control much more what's happening.
So I think there's a lot of use cases.
I think that you need to have local inference.
But at the same time, there will always be the cloud, the big model inferences as well.
And local inference generally is not really like a big compute problem, but more like a memory bandwidth problem, isn't it?
You just want to have unified RAM and launch bandwidth and that's it, right?
Yes, I think that because the code step in current LLM paradigms is like you need to load entire weights in each forward path into the memory, into the compute units kind of, and that creates a lot of, as you said, memory bandwidth bottleneck.
There are a lot of works to kind of resolve these issues.
There's different non-autoregressive LLMs.
There's also a mixture of experts and so on that actually only activate certain parameters and so on and sparsity specifically as a way to kind of reduce that.
But then also, for instance, with MOE, it's quite interesting because you can make a very sparse model that's actually still very good, but then the total memory will be very big.
And that again, like we mentioned, a phone that has only like, or a laptop that's maybe 16 gigabytes or something like that.
And the 16 gigabytes...
actually, you want to use it for your browser, for your operating system.
So the amount of memory that you can actually dedicate on your LLM is actually very small.
So there's always this trade-off.
I think there's like a, or the way I see this line of this triangle where you have compute, which is this flops.
And all of the talk, I think, or 99% of the talks that you see is about flops, is about compute.
And this is definitely one aspect as well, but I think it's only one part.
The other two kind of edges on the triangle is like memory bandwidth, as mentioned, because like, you know like you a raspberry pi for instance has actually quite a lot of ram given its compute but very slow bandwidth so you can actually like to get this 42 seconds per second as i mentioned before is quite challenging in terms of the memory bandwidth and then also memory size because like um you know the the you know as i mentioned the phone the phone kind of has like a decent memory bandwidth like you're talking about 70 gigabytes per second and so on but you know the total amount is very limited.
And then the operating system, the browser and so on, they actually consume a lot of memory.
So you have actually very little memory available as well.
So these are the three kind of edges on the triangle that determine what model we can run.
And do you see that changing potentially?
I mean, A, like memory optimization of LLMs and B, basically larger memory and phones, etc.
And larger bandwidth?
Yeah.
For instance, one area, as I mentioned before, the flops, I think there's this neural processing unit, the NPUs, it's kind of a special chip area on the chip that's just very fast for this low precision computation that's actually needed by the LLMs.
And I think this is actually where a lot of the benefits are out there.
So there's like, you can, let's put it this way, the amount of operations that a phone can run and more than one, it's actually very, very.
fast.
So what I will see probably in the future is to kind of leverage that amount of flop availability while, you know, memory shortage at the moment, right, that we are in.
And also memory bandwidth that evolves much, much slower.
So I think what we'll see in the future is much more compute heavy, but memory friendly kind of approaches, you know, where you kind of offload a lot of the stuff that's like reusing memory, things like that.
I think there's much more creative solutions that we'll see.
takes, as I mentioned before, the hardware in mind, where the hardware is evolving, you want to have access to the latest hardware, such as the next generation hardware, such as you can already design the model in according to the profile of memory, flops, and memory bandwidth that you have.
Okay.
And from your perspective, how much graph number, how many tokens per second does a phone really need for an assistant nobody noticed that it's local?
It depends on the task, I would say.
Like there's certain chat applications.
And if you draft, for instance, a message, you want to actually be very fast, like 100 tokens per second, those kind of 80 plus 200 tokens per second.
A lot of background tasks and so on that you can probably get away with like 20 or so.
So that's a lot of use cases.
There's also this kind of control.
There's this computer use where you just basically tell or speak what the phone should do.
where you actually want to have a lot of tokens per second because you actually want the more phone to take five minutes to kind of open an app and call an Uber or something like that, right?
Yeah.
Okay.
Yeah, makes sense.
So basically, a 1 billion parameter model will never be as smart as a 3 billion parameter model.
And the answer is basically that you have to have very good tools and retrieval.
I assume.
So is that the local bet really about tools then?
And having the right tools at hand for a model and then really have it quickly resolving to a tool and running that to basically simulate better quality?
I think it depends on the task.
I think that's like...
I can't say too much because we're working with some of these companies kind of the way we're working on phone use and so on.
But there's, I would say, not one super app kind of model, but really small.
Basically, think of small models that kind of nudge here and there for notifications and so on.
I think we want to hand over this entire control to a model to an agent, as mentioned before.
So I think there's a lot of use cases in there that you can actually make the phone more intelligent and smarter.
Yeah, I think, I mean, I run my local home assistant with a bit of AI and that like there, it's really like having the tools at hand and the tools available really makes the model powerful then, right?
And really shrinking that tool context also to provide it with the right tools.
And then, yeah, but it's still a bit clumsy, to be honest.
Yes, I can imagine.
So what do you think like in 2028, what runs on devices and what runs on the cloud.
So I guess the typical Siri use case will still continue local and other things like the bigger cases will run remotely, run in the cloud.
What about small agentic tasks?
You mentioned that people were trying your model as a coding model.
And I think many agentic tasks are actually...
building code and then executing that, right?
So does that mean that this is only for the big models or how does that look like from your perspective?
Or is that like specific models for just running tasks in certain languages that are really compact, etc.
Or how does that look like?
Yeah, I would say to answer the thing with the coding agent, we pre-train also our model on a lot of code.
as you said, because a lot of the genetic use cases, it's just you do the model, right?
So like a bash script that executes that, right?
You want to pipe it to another command line and so on.
So there's like a lot of need for kind of this low, like glue code or so it's called, I think, where you just pipe the output from one to another thing and so on.
Maybe processes and so on slightly.
I think there's like a lot that we will pack into the model as well.
It becomes a bit...
tricky with like permission and so on right there's i think same sandboxing and so on problem that we have out there that you want to you know you ideally want to give the model access to kind of a bash or like a general purpose shell but at the same time it's also very dangerous if you just let it loose in a way and you need to sandbox it and so on so i think i'm i don't have an answer to be honest on this um i think we'll see like in the future of what you know prevails there's like on on one hand like this kind of you want to give the model general purpose capabilities because you want to like unrestrict the use case you know as much as possible but at the same time you know like you have a lot of like you know when it goes wrong in the wrong direction so on maybe this is just a very well built sandbox and so on and you want to give still the coding agent like a small coding agent in a way um that could be the solution um but we'll see but but one one other things that as i mentioned before like i think It's a difference between writing a bash script or a glue code that you just exude as part of the agentic task, or really like coding, like you build a web application kind of level of coding.
I think for the web application level of coding, you will probably want to use the big model.
And for the small tasks that will most likely run local as well, right?
Exactly.
For the small tasks that run local, we want to have latency.
You have this, what a lot of these companies that work together call unmetered.
intelligence or you kind of instead like you imagine you have free tokens you want to token max and so on like there's a lot of this talk about right now out there right about this token max or like the amount of you know they spend so much yeah and if you have a good local agent you can token max because it's like it's basically free kind of yeah free as energy costs etc right but absolutely so just out of curiosity like I mean, you said you're mostly doing research, but how does your local setup look like?
What do you use day to day?
I guess you're also partly coding.
How does your setup look like and how careful are you actually with agents?
Because you just mentioned generating bash scripts is also pretty dangerous.
like how how how how paranoid are you yeah that's like my my setup i think is usually like i think three different coding agents like i'm using like cloud code a lot i'm using codex as well a lot and also like open code and i think that people will ask like why these three different harnesses um it's because i want to like stay up to date of what's happening so it's not just i'm not a consumer only but i also want to say hey how much how much better it is, you know, the current cloud model compared to like an open source Chinese model from two months ago.
Is there like a noticeable difference, you know, a lot of the internal tasks and so on to kind of see where we're going, you know, to kind of get a feeling.
So that's why maybe I'm over exploratory on this.
And as I mentioned, or as we talked before about the sandbox, I'm actually super annoyed by all of the permission, you know, requests and so on.
So I'm usually running this in a YOLO.
kind of fashion where you just...
Good.
But would you give it access to any personal information?
Would you give it access to emails?
Like how many MCPs do you run locally?
Like is it your personal assistant as well?
Or how far do you go there?
Yeah, actually with my email and so on, like very little.
Like I'm still doing a lot of this manually and so on because a lot of like privacy reasons or other reasons and so on.
But like with just...
you know bash executions on the code i would say mostly yolo kind of you just want to let it cook there in a way um to not like approve every single change um yeah like i think it's it's a it's a well-balanced question i think because at the same time you you hand over a lot of information about the company and so on about your own company upon your not just privacy but also like the company information so on tool like some of these providers So there's always, you know, like, I don't want to like, you know, did some of these calling agents companies, I think probably have access or like at least seen so many internal details about all the kind of companies worldwide.
So I think that's also a bit of like, you know, always like a thing, like I'm not sure whether this stays that way or this will be the default or this will, you know, people will move to like local or like on-prem setups kind of as a hybrid.
So it's a bit of, I don't know, a few years ago when people were publishing photos on Facebook without being afraid.
So to come to your main case, which is cars right now, what does run in cars in Mercedes CLA, for example?
What do you call there?
Is it like the lane keeping and self-driving or like in-car assistant or what is it?
What your model powers?
It's in-car assistant.
And the reason is also like we choose, because we get also a lot of requests for autonomous driving, you know, set up and so on from very, very different automotive automotive companies and so on.
But I think we are very cautious about it because the time it takes to deploy, to certify and so on is so long.
So as a startup, we just want to get the quicker wins first.
So this is why we focus also on the assistant, because there's a lot of...
The current solution that you see in all of this voice assistants, and this is not just cars, but I think in general, like devices, smart devices in general, is that they're very rigid.
They're like pattern matching and so on.
They don't really feel intelligent.
They're feeling more like your pattern match to kind of a previous request or so, or like the developer, like your mates mid.
intentions in a way that the developer designed to have.
And if you ask anything outside or if you just phrase it a bit differently, it completely trips over the system.
And so that's why with an LLM-based voice assistant and so on, you can actually build some intelligence in there that kind of interpolates between the different ways to phrase it.
Also more like subtle and so on.
Like if you have a conversation and you just say like, hey, I'm cold or so that you find the car to text that, hey, I should actually increase the temperature and those kind of things.
Well, if it's part of a context and so on, again, the LLM should be able to parse this correctly.
Are there like tons of, I don't know, simple Chinese models available that manufacturers also could just choose?
Or is it also like basically having control and trust over the end product and efficiency, as you said, like of inference?
Or what makes them switch to your solution?
yeah there's actually very good or omni or audio models out there as well as you mentioned from the chinese and sound like i think a coin 7 billion parameter model and so on it we tested in past it seems to be very good um these are absolutely not able to run locally on the hardware that some of these companies have you you we're talking about here like um hardware that's i would describe a raspberry pi level you know like really like limited in terms of compute.
So like a 7B model would run there maybe with like half a token per second or so.
Okay.
Whereas you need much quicker.
So I think like the uniqueness of Liquid is really that we did not just build a model and like that's small and therefore it's fast, but like for us, speed is non-negotiable.
We start actually with the speed.
We start with speed in mind, like what's the hardware can give us.
Then we kind of try to kind of get the capabilities into this small footprint versus like for a lot of open source models from China, it's just 7B is just like Lama decided, or even before OpenAI decided to train a 7 billion power meter GPT-3 model.
And this has to be kind of standard, but there's no reason why it should be 7B, right?
It's more like a research artifact.
But for us, it's different.
We start with the speed, we start with the hardware, our first point.
And you start with assuming that there's very limited hardware and cards available always.
Why is that the case?
Because of energy consumption or just because car industry is a few years behind and very conservative and very safety focused?
There are several reasons.
I think also the amount of diversity that I've seen in the cars is quite a lot.
You see basically gaming consoles level of computing the infotainment unit.
You see like Raspberry Pis kind of powering the infotainment unit.
So you have seen a lot of difference.
So there's not one standard.
But it's very different.
And then one other aspect is that you, similar to what I mentioned before, the form is that the LLM or this system is not the only thing that runs on this device.
But there's like a thousand other things, the navigation system, the multimedia system and all of this stuff that runs as well.
So that's why you actually get only a small footprint or a small slice of the entire budget.
And the slice could look like a Raspberry Pi, for instance, even though the entire system...
is actually quite powerful.
Looking at the model longevity, so a car typically lives 10 years and a model lives roughly six months or so.
Is the plan then that you update your next model also to all existing cars again?
Is that basically a subscription model for the car manufacturer that they always get an update?
Or how does that work?
And is your goal then to...
in 10 years still run on those old cars or?
Yes, that's, I would say like depends on each, like at least how it's done right now.
Like it's very, you know, like a case by case basis.
But our vision, this specifically also like when you, when you, when people speak to Ramin and so on or now speak with Ramin and so on, is that we want to build like a continual learning platform.
Because the thing that you mentioned, like every six months you have a new model.
our own models you know we have some stuff that we hopefully release in the next couple of days and so on next couple of weeks that's also very exciting and so on so there's always you know change and then some of the deployments some of this qa just quality assurance you know of this process of this company takes like uh much longer than our development cycle so you can imagine that by the time it goes public life it's this it's already out there and you know in a way to solve this problem we're working on a platform kind of that is powered by agents and kind of because as i mentioned before like this cloud code and a lot so a lot of these agents can actually do a lot of work So why not also making sure that when Liquid builds a new model, a next-generation model, that the customer gets automatically an upgrade of their own, the current production model, the same data, same evaluations have been applied to a new generation model.
And so basically you get just notification.
And you're just similar to how you upgrade your laptop.
You just click accept.
I mean, it's the simplest scenario.
Obviously, there's much more testing and so on necessary.
Our goal is to build a platform that kind of allows updating models much quicker than it is currently feasible.
So in a few years, you'll have a parking lot full of old or various different cars where you can run tests continuously and check if the model still runs fast enough, etc.
Yes.
That's the plan.
Good plan.
You're also releasing your models as open-weight models, right?
So everyone can use them, test them.
Any advice?
If any of my listeners want to test your model, where would you start?
What would you advise to do?
What would you advise to test with them?
I think downloading them.
I think there's a lot of pre-packaged GGUF, like Falama CPP.
quantized models, you know, that are actually quantized.
And the way we quantize it is also like we do a lot of like quantization aware training and so on, where you actually prepare the model to kind of run with low precision during the training.
During the training, you are the model, like some of this low precision that are inference and so on.
So these models are out of the box, quite performant, you know.
Obviously they're small and you cannot expect the same uses as I mentioned before.
Like you don't want to like build a web app with like a small, you can, but I would not recommend it in a sense.
But just test it in terms of the speed of what they can do.
Also, you can enhance the capabilities with constraint decoding.
So imagine you have to chase an output that you enforce.
You can make it much more robust, right?
Because from out of the box, the model may in 9 out of 10 cases succeed, but in one case it fails.
But if you do constraint generation, there's also a lot of fine-tuning.
And with lower rank adapters, fine-tuning is actually very cheap, which was in the past always a problem and so on.
So I think it would be nice just to test the model, specifically with this quantization aware trained model, which is just observe the speed and capabilities you can see.
I'm actually quite impressed by the amount of things that you can achieve.
Obviously, as I mentioned before, there's a caveat.
These are small models that you need to find in them usually and so on to kind of boost the quality, have other galleries around it.
But I think it gives a good sense of like, hey, where are we going?
So test it on your Raspberry Pi.
That's your recommendation.
Great.
And I'm super curious how actually this whole training setup looks like you have there.
I guess that's also part of your mode, right?
And obviously a big investment.
How big is your server park around this?
Is it all your hardware?
Where do you get the GPUs from?
How much do you need?
How does the training run work?
I don't know how much you can say about this, but I'm super curious how this actually looks like because all folks I know are mostly doing inference, right?
And no one does training or less training, I would at least say.
How does this look like?
Can you give us a rough insight on that?
Yes, Lister, as far as, yes, I can try to give you a level of view.
At least, yeah, we have like a couple of thousands of GPUs.
So that's roughly the ballpark of what we're talking about.
But there's not like one big training run that you just use 10,000 GPUs or to train one model.
But, you know, there's a lot of things going on.
Like there's customer deliverables that use a certain of them.
You know, there's maybe one big training run that use half of them or something like that.
There may be some other things like multimodal.
There's also research, as I mentioned before.
We are trying to look into what comes after autoregressive LLMs and so on.
A lot of work is going on with things.
So there's a lot, I would say, our cluster is quite diverse.
So there's not one big cluster that you do only pre-training and so on.
But it's really much more, I would say, not boring, but much more like an academic cluster.
So if people come from academia, you know, they have, at least when I did my PhD and also like at MIT and so on at the postdoc, you get access to kind of the cluster and like use of new jobs.
And it looks very similar, I think, to kind of that the workflow is not that, you know, like something with the pre-training that's magically different.
It is probably the data compute, you know, the amount of compute that some of the training jobs and so on are actually bigger.
That's true.
But like, I think it feels very academic, I would say, like, especially like going from PhD postdoc and then, um to to the way we currently train the models um it feels very seamless to be honest maybe maybe maybe it is because like we made the cluster kind of and we decided the way we were used to it but at least that's that's how we run it at liquid so i would say like a person who does academic work research work it feels very like flat almost no change at all but just bigger And who does the dirty work of getting all the data?
I mean, data partnerships, the scraping, etc.
To a large extent, this also belongs to a training run, right?
And how does that look like for you?
And how do you think that looks like in other labs?
Especially the Chinese ones, I don't know.
I mean, there has been a lot of talks about where they get the data from and then everyone steals from everyone.
How does it look like?
Data is such an important aspect of ML.
So that's why, for instance, even myself, I was involved in our trader, how we obtain it, how we get data, up to very recently.
Where I'm still actually involved in a lot of extent because it has much more impact or much more, but it has such a big impact on the model quality that comes out.
And at the beginning, when we started, we were scraping from what's out there on the web, a lot of Hugging Face datasets.
Luckily, there's a lot of Hugging Face datasets out there that are very good.
And most of the work, to be honest, was not necessarily crawling, but really of people that have crawled and uploaded data on Hugging Face to do quality assessment.
Specifically, there's a lot of, I would say, garbage or trash on Hugging Face.
People that just crawl and they don't really...
properly filtered and so on, you have massive garbage out there.
And there's some data sets that are really carefully crafted, you know, and a lot of work to be honest was less, as I mentioned on, on crawling itself, but really like going through data set and like looking into like, Hey, the people who created this data set, like, is this actually wealthy that can be trained, can we use this in our training set and so on?
So it's, it's, it, it is kind of boring in a sense that you have to look at it a lot.
But it has such an important impact on the model quality that it's something that you cannot automate.
You try to build a classifier and the classifier fails because it's now biased in some extent because you only have such a small labeled set and so on.
So it's quite interesting the amount of stuff that I've seen in some of these very well-filtered public datasets out there.
It's crazy.
Some of them, as I mentioned before, are super very well-curated.
even though they have a lot of citations or a lot of reach, they are very poor in terms of quality in a lot of aspects.
Okay.
And how do you assess the quality?
Is it then really testing that, evaluating it against assumptions or not assumptions, but things you know that they are right?
Or how does that look like?
Here, I think that...
The advantage is that I did a lot of machine learning classes when I was doing my PhD and so on with very good professors.
I think I have deep respect for them who really like going to this kind of training and test distribution.
And the LLM space, a lot of things like it's AGI, it's magic and so on, but it behaves very, very similar to kind of the boring kind of machine learning problems that people told in their classes and so on.
And if you look at it really from a test distribution angle, if you say basically what it means, if you have a data point, the question that I have is, would a user of our models ask something like that?
Does this sample, in a way, that's the sample that I'm looking at right now, does it provide information?
Does it provide value to how people end up using our model?
The thing about this distribution stuff that I mentioned before, it doesn't fall into my test distribution or my usage distribution and so on, or is it just some random stuff and so on?
And it basically could change a lot based on how the model is used.
So if you're trained, as I mentioned before, the coding model and so on, you want to have much more coding data, even authentic now.
Even if you want to crawl from GitHub, that's how basically people did a lot of coding models in the past, but now actually you want to do much more authentic.
passively observing code, maybe not actually the ideal strategy anymore, but you want actually to modify the code you want to write and so on.
So there's a lot more like the connection of code with tool, you know, that modify or if you were like operate on this code is much more valuable, it's much more in distribution to how people use the model than just passively observing the code, for instance.
And I think that's how a lot of our team members, you know, look at the data problem.
Okay.
And so that means that you basically have your specific use cases also as test cases and then run against them.
So what does it actually do if I, or what actually comes out if I tell the model that I'm cold, right?
Like what does it do?
Does it open the window or so?
I mean, that's the obvious cases, but...
If you really want to train a large language model, then there's still a very large gap between this and a few billion parameters model, right?
How do you come from your typical use cases to the more generic things?
And how do you then...
Or how does it look like?
Do you just build your test sets for ages?
How do you extrapolate from one to another?
As you said, because the harm and cold use case is very specific and so on, and how do you determine this?
I think instead of using the downstream usage, what we use instead is these evals.
And here we have the evals.
We created some of them ourselves, some of them public, and they are kind of a reflection of potential use cases.
Like you think of a translation model, think of just an agent where you have like tools and you have like, hey, you're controlling a machine or you're controlling a room temperature stuff or like the HVAC system of a building or something like that.
And then the agent needs to do like the right things and so on.
So you build evals kind of, or use public evals kind of to approximate.
how people downstream will use the model and then you build kind of the data for those benchmarks in a way so for instance like in if you if you care about you know you know a lot of like to give you an example a lot of this kind of benchmarks and you and so on that you see mlu pro a omniscience and so on is a more modern version of that because like they are usually outdated also as quickly as the models get released They are very heavily STEM-focused, like science, technology, engineering, and mathematics.
So basically, if you have pre-training data on those topics, on science and so on, it will show up in the evaluations that your model has captured the knowledge from these science textbooks or science topics that you have in the pre-training set because it will improve the knowledge, understanding of all the science topics, and it shows up in the benchmark.
and then downstream how this will be used and so on.
Okay, understood.
Yeah, a little intro to model training and evaluation.
Thanks a lot for that.
Any great talks around the topic you saw recently that are really eye-opening and that could help others that want to really dig deeper on the topic?
Anything to get started with where you think, hey, this is brilliantly explained and brilliantly teached?
Anything you can recommend?
Any talks?
That goes a bit into marketing materials, at least for my colleagues.
One is from Alexander Amini.
He has this course on deep learning, internal deep learning, which is very nice.
I think if people want to go into deep learning, there's also from Maxime Le Bon, our head of posturing, who wrote the LLM engineer handbook.
And it's really about prompting, about fine-tuning, about data and so on that we discussed.
And I think it's a good overall, if you want to engineer an application with LLMs.
like what topics and so on, what the pitfalls are there.
So I think these are very good materials.
But in general, the space is moving very quickly.
So for instance, even this textbook and so on, maybe a lot of agenting stuff, a lot of multimodal aspects get outdated, I think, very quickly.
Which again, I think leads a bit to this kind of in the future outlook, I think, into looking to Omnimodal, like video and audio input.
I think it will become like...
as standard as currently text-only LLMs are.
You said you're basically watching, observing all the big models, using everything in your harness locally.
Can you tell us about your recent aha moment when even you working in AI really had a moment of shiver?
What was it?
That's difficult because there's a lot of things, a lot of excitement.
Some of them I expected, but one thing that I think specifically is interested in would be tiered memory systems.
And specifically what I mean is that we talked about this memory bandwidth or memory size as a capacity limit.
So I think designing models kind of not just have like one pool of memory, but they maybe have like a faster memory that's smaller and a slower memory that's bigger and so on.
I think because that's, I think, standard in computer science, if you think about it, like a CPU as a cache.
Then you have your memory, you have your storage, you have your internet, you have your bandwidth and so on there.
So I think computers in general work on a tiered storage system.
And I think there's a lot of works going into the direction of like, hey, how can you build a model that kind of taps into this tiered memory system?
So that's, I think, for me, very exciting because there's a lot of untapped potential.
So basically organizing memory, organizing context, etc.
and really limiting that so that you can still be efficient and that you can focus on a narrow task, right?
Any recent movements you saw there that were really surprising for you?
I think there's not one thing, I think, but there's just a lot.
There's a lot of talks that people connect, I think, the SSDs to the data center GPUs.
and things like that.
There seems to be a lot of very interesting ideas going around.
Okay, yeah.
Let's see where we land, right?
Let's see where we land.
So we are slowly approaching to the end and I still have a little surprise for you.
So I actually found an old lab machine from TU Vienna with a home simulation still running on it.
And it had 302 neurons, 8,000 synapses, and there was a hidden flag in the config, which was basically like a minus-minus year you could run when you evaluate it.
And we now, assume we now run it with the year 2017.
And it basically moves us back to your old Radu Grocers Lab.
And your master student staring at a wall.
And we now can watch Matthias for a while.
And you would now have the chance to whisper something into young Matthias' years.
What would you whisper to yourself if you were in the year 2017 again?
That's a good question.
Because back then I was quite excited about reinforcement learning.
because it was just OpenAI, released the OpenAI Gym and PPO, the paper, the Prophexional Policy Optimization paper, and I was like super hypey and so on with AlphaGo and so on.
But then I think it died down a bit and now it's like back.
So I would basically tell myself in the past that they probably stick to it.
Like it will fail, but it eventually will.
Don't give up reinforcement learning.
Okay.
Cool, cool.
Anything else?
Anything else?
I mean, buy Bitcoin, obviously, etc.
Like all those traditional things.
But I think generally the topic of hardware awareness, I think that's a lot of the stuff also that we did, they are very small and inefficient.
But even though it's efficient, it doesn't matter if it's not efficient on the hardware that we had available.
if that makes sense right the gpu like like we had this problem actually back then already like we have like 19 neurons and like a gpu has a 3 000 cores right so it's a question it's like how do you distribute it what look so a lot of a lot of the hints or tips i would give myself is probably like hey think about maybe deeper on like okay hardware awareness you know that you have like efficiency maybe you need a different type of hardware you know and where you can find this or um if you have like gpus available like how can you actually change the models to leverage the parallelism that you have there.
Okay, cool.
So focus on inference efficiency straight away.
So Matthias, really great talking to you and really looking forward to hear who else will use your models in their cars and will actually now in the next hour run your model on my, not Raspberry Pi, but my local setup here.
and try it out with Home Assistant.
I guess it can do tool use, right?
Obviously.
Cool.
Especially if you use latest models, yeah.
Okay, cool, cool.
We'll try it out.
Thanks a lot for the talk and best of luck for your future and your models in the future.
Let's see where we see them in the future.
Thanks a lot.
Thank you so much.
It was great talking to you.
Bye.
Thank you for listening to the Alphalist podcast.
If you liked this episode, share it with friends.
I'm sure they love it too.
Make sure to subscribe so you can hear deep insights into technical leadership and technology trends as they become available.
Also, please tell us if there is a topic you would like to hear more about or a technical leader whose brain you would like us to pick.
Alphalist is all about helping CTOs getting access to the insights they need to make the best decisions for their company.
Please send us suggestions to cto at alphalist.com.
Send me a message on LinkedIn or Twitter.
After all, the more knowledge we bring to CTOs, the more growth we see in tech.
Or as we say on Alphalist, accumulated knowledge to accelerate growth.
See you in the next episode.