Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola 2026-07-27T00:00:00+00:00 https://blog.none.at/tags/web-scraping/atom.xml 2026-07-27T00:00:00+00:00 2026-07-27T00:00:00+00:00 aleks https://blog.none.at/blog/2026/2026-07-27-istio-envoy-ai-bot-blocking/

<p>Blocking a list of known AI-crawler user agents at an ingress gateway sounds like a five-minute job: match the <code>User-Agent</code> header against a list of bot names, return <code>403</code>. The obvious tool for that on <a rel="noopener external" target="_blank" href="https://istio.io">Istio</a> is its own <code>AuthorizationPolicy</code> CRD — until you actually try to express “header contains one of these ~160 substrings” in it, and discover the API simply cannot say that. The next obvious fallback — a Lua filter that runs the match by hand — works, but is more machinery than the problem needs. There is a third option: configure the same native Envoy filter Istio’s own CRD compiles down to, directly, with the one matching primitive the CRD never exposes. Getting there also surfaces a second, unrelated gotcha in how Envoy’s regex matching actually works, which would have made an obvious “just use one big regex” approach fail silently.</p>