Alek's Blog

1 post with tag #autoscaling

Diagram: Prometheus, Grafana, and KEDA observability stack for LLM inference
vllmllmkubernetesovhgpuprometheusgrafanakedaobservabilityautoscalinginference

LLM Inference on OVH MKS: Prometheus, Grafana, and KEDA

Scrape vLLM and DCGM metrics with kube-prometheus-stack, visualise TTFT and tokens/s in Grafana, and autoscale to zero with KEDA. Part 4 of 6.

LLM Inference on OVH MKS: Prometheus, Grafana, and KEDA