<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
    <channel>
      <title>Alek&#x27;s Blog - web-scraping</title>
      <link>https://blog.none.at</link>
      <description>Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty.</description>
      <generator>Zola</generator>
      <language>en</language>
      <atom:link href="https://blog.none.at/tags/web-scraping/rss.xml" rel="self" type="application/rss+xml"/>
      <lastBuildDate>Mon, 27 Jul 2026 00:00:00 +0000</lastBuildDate>
      <item>
          <title>Blocking AI Crawlers on Istio: AuthorizationPolicy&#x27;s Ceiling and Envoy&#x27;s Full-String Regex Trap</title>
          <pubDate>Mon, 27 Jul 2026 00:00:00 +0000</pubDate>
          <author>aleks</author>
          <link>https://blog.none.at/blog/2026/2026-07-27-istio-envoy-ai-bot-blocking/</link>
          <guid>https://blog.none.at/blog/2026/2026-07-27-istio-envoy-ai-bot-blocking/</guid>
          <description xml:base="https://blog.none.at/blog/2026/2026-07-27-istio-envoy-ai-bot-blocking/">&lt;p&gt;Blocking a list of known AI-crawler user agents at an ingress gateway sounds like a five-minute
job: match the &lt;code&gt;User-Agent&lt;&#x2F;code&gt; header against a list of bot names, return &lt;code&gt;403&lt;&#x2F;code&gt;. The obvious tool for
that on &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;istio.io&quot;&gt;Istio&lt;&#x2F;a&gt; is its own &lt;code&gt;AuthorizationPolicy&lt;&#x2F;code&gt; CRD — until you actually try to
express “header contains one of these ~160 substrings” in it, and discover the API simply cannot
say that. The next obvious fallback — a Lua filter that runs the match by hand — works, but is
more machinery than the problem needs. There is a third option: configure the same native Envoy
filter Istio’s own CRD compiles down to, directly, with the one matching primitive the CRD never
exposes. Getting there also surfaces a second, unrelated gotcha in how Envoy’s regex matching
actually works, which would have made an obvious “just use one big regex” approach fail silently.&lt;&#x2F;p&gt;</description>
      </item>
    </channel>
</rss>
