Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola 2026-06-27T00:00:00+00:00 https://blog.none.at/tags/gateway/atom.xml 2026-06-02T00:00:00+00:00 2026-06-27T00:00:00+00:00 aleks https://blog.none.at/blog/2026/2026-06-02-llm-inference-on-ovh-gateway/

<p>Parts 1–5 deployed a functional self-hosted LLM endpoint. This part adds <a rel="noopener external" target="_blank" href="https://docs.litellm.ai/">LiteLLM</a> — an open-source proxy that exposes a unified OpenAI-compatible API across multiple LLM backends — in front of vLLM. The gateway layer brings per-user API keys, budget enforcement, and automatic fallback to commercial APIs when the local model is unavailable.</p>