https://blog.none.at Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola en Sat, 27 Jun 2026 00:00:00 +0000 Tue, 02 Jun 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-06-02-llm-inference-on-ovh-gateway/ https://blog.none.at/blog/2026/2026-06-02-llm-inference-on-ovh-gateway/ <p>Parts 1–5 deployed a functional self-hosted LLM endpoint. This part adds <a rel="noopener external" target="_blank" href="https://docs.litellm.ai/">LiteLLM</a> — an open-source proxy that exposes a unified OpenAI-compatible API across multiple LLM backends — in front of vLLM. The gateway layer brings per-user API keys, budget enforcement, and automatic fallback to commercial APIs when the local model is unavailable.</p>