LLMD reaches 1.0 for distributed LLM inference

LLMD, a project for distributed inference on Kubernetes and part of the vLLM family, recently reached version 1.0 and is described as production-ready. The project is donated to the Linux Foundation and is being released as part of Red Hat's product portfolio. LLMD addresses the limitations of standard Kubernetes scheduling for LLM inference by managing stateful conversation contexts, specifically maintaining KV caches and prefix caches to ensure requests stick to the same vLLM instance for efficiency.

The system introduces batch inference capabilities to utilize idle cluster capacity for background processing, aiming for 100% hardware utilization. It supports automatic flow control, quality of service prioritization, and observability metrics for token production and throughput. LLMD can run on bare metal as well as Kubernetes and is designed to maximize the value of expensive GPU hardware from vendors including NVIDIA, AMD, Intel, and Google TPUs.

Red Hat argues that LLMD allows companies to own their AI infrastructure, enabling on-premises inference without leaking data to third parties while maximizing the return on investment for expensive GPU hardware. A representative noted, "LLMD is not just a router or a scheduler. It's actually a family of projects, and they are all meant to give you a set of tools to do distributed inference so that you can leverage all of the GPUs that you have and use it at their maximum."