Adaptive Load Balancing for Distributed LLM Inference Using Ollama

Abstract

Large Language Models (LLMs) have become crucial components of modern AI systems, supporting a wide range of applications through natural language generation tasks. While open-source serving platforms such as Ollama simplify the deployment of LLMs in local and on-premises environments, efficiently distributing inference workloads across multiple Ollama instances remains an open challenge. Conventional load balancing strategies, initially designed for stateless web services, fail to account for the dynamic execution characteristics of LLM inference, leading to suboptimal resource utilization and increased latency, particularly in heterogeneous computing environments. This paper proposes OllamaRouter, a specialized load balancing strategy tailored for distributed LLM inference using the Ollama text generation API. The proposed strategy dynamically allocates requests based on runtime estimates of token processing time and queue load, adapting to heterogeneous node performance without introducing significant computational overhead. To evaluate the effectiveness of OllamaRouter, we conducted a controlled experiment comparing its performance agains conventional Round Robin and Least Connection strategies, under a constant incoming request rate. The results show that OllamaRouter delivers higher throughput and lower average request latency, particularly as the number of waiting requests increases. The observed practical improvements and increased stability highlight the potential of adaptive load balancing in optimizing distributed LLM serving scenarios.