Optimizing the LLM Inference Stack
28 min read Systems Performance EngineeringAn end-to-end framework for optimizing LLM inference: map hardware, kernels, host execution, runtimes, and orchestration, define service demand, rank bottlenecks with capacity and queueing, and choose experiments that improve goodput. Includes case studies from Makora and Baseten.
Tags: performance, systems performance engineering, system design, inference serving, benchmarking, queuing theory, kubernetes, linux, gpu, hardware, llm-d