https://blog.none.at Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola en Sat, 27 Jun 2026 00:00:00 +0000 Thu, 14 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-security/ https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-security/ <p>Access control, encryption, and immutability requirements often determine whether a log archive can be used for compliance purposes — and the five systems differ substantially in what they enforce natively versus what must be delegated to infrastructure.</p> <p>This is Part 3 of a five-part series.</p> <ul> <li><strong>Full guide:</strong> <a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-guide/">Elasticsearch vs. OpenSearch vs. Loki vs. Quickwit vs. ClickHouse: The Complete Guide</a></li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse/">Part 1 — Comparison:</a> Storage tiering, compression, resource consumption, query languages, SaaS options</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-operations/">Part 2 — Operations:</a> Setup, ingest pipeline, ECS vs. OTel, backup &amp; DR, observability, alerting</li> <li><strong>Part 3 — Security &amp; Compliance — this post:</strong> Encryption, RBAC, WORM / S3 Object Lock</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-ux/">Part 4 — UX, Dashboards &amp; Alerts:</a> UI layers, log search UX, cold-tier query behaviour, dashboard building, sharing, alerting</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-usecases/">Part 5 — Use Case:</a> Vector, Istio/Envoy, nginx, Java on OVH MKS — complete ClickHouse ingestion setup and monthly awffull report</li> </ul> Thu, 14 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse/ https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse/ <p>Long-term log archiving is a deep topic. Each of the five systems covered here has its own tiering model, object storage integration, query language, operational quirks, and security posture. To do it justice, this comparison is split across <strong>five posts</strong> — take your time reading them, and don’t hesitate to come back to individual sections more than once.</p> <ul> <li><strong>Full guide:</strong> <a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-guide/">Elasticsearch vs. OpenSearch vs. Loki vs. Quickwit vs. ClickHouse: The Complete Guide</a></li> <li><strong>Part 1 — this post:</strong> Storage tiering, compression, resource consumption, query languages, SaaS options</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-operations/">Part 2 — Operations:</a> Setup, ingest pipeline, ECS vs. OTel, backup &amp; DR, observability, alerting</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-security/">Part 3 — Security &amp; Compliance:</a> Encryption, RBAC, WORM / S3 Object Lock</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-ux/">Part 4 — UX, Dashboards &amp; Alerts:</a> UI layers, log search UX, cold-tier query behaviour, dashboard building, sharing, alerting</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-usecases/">Part 5 — Use Case:</a> Vector, Istio/Envoy, nginx, Java on OVH MKS — complete ClickHouse ingestion setup and monthly awffull report</li> </ul> <hr /> <p>Storing logs for compliance, auditing, or forensic purposes often means retaining data for <strong>seven years or more</strong>. That is a very different problem than storing the last 30 days of operational logs. Hot-tier query latency matters less; cost per GB, the ability to still search old data without restoring it manually, and the operational burden of keeping the system alive for years become the dominant concerns.</p> <p>This post compares five systems — <strong><a rel="noopener external" target="_blank" href="https://www.elastic.co/elasticsearch">Elasticsearch</a></strong>, <strong><a rel="noopener external" target="_blank" href="https://opensearch.org">OpenSearch</a></strong>, <strong><a rel="noopener external" target="_blank" href="https://grafana.com/oss/loki/">Grafana Loki</a></strong>, <strong><a rel="noopener external" target="_blank" href="https://quickwit.io">Quickwit</a></strong>, and <strong><a rel="noopener external" target="_blank" href="https://clickhouse.com">ClickHouse</a></strong> — along the axes that matter for long-term archiving: tiering, object storage integration, compression, resource consumption during ingest/compact/search, and available SaaS options. A separate section covers systems that were evaluated but are <strong>not recommended</strong> for this use case.</p> Tue, 12 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-12-nfs-sync-via-rclone/ https://blog.none.at/blog/2026/2026-05-12-nfs-sync-via-rclone/ <p>I needed to sync and back up an NFS share to cloud storage for example to Azure Blob Storage — reliably, with a hard “no version must ever be lost” SLA, and for a file tree that could grow to 1M+ files. This post documents the architecture decisions, the core sync script, the lab setup for testing without real NFS or Azure access, and the packaging for RPM, DEB, Nix and Alpine. The sync engine is <a rel="noopener external" target="_blank" href="https://rclone.org">rclone</a>.</p> <p>The result is <a rel="noopener external" target="_blank" href="https://github.com/git001/nfs-sync-via-rclone">nfs-sync-via-rclone</a>.</p> Thu, 12 Mar 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-03-12-blog-log-shipping/ https://blog.none.at/blog/2026/2026-03-12-blog-log-shipping/ <p>This is a companion post to <a href="https://blog.none.at/blog/2026/2026-03-12-blog-awffull/">AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs.</p> <hr /> <h2 id="shipping-access-logs-to-object-storage">Shipping Access Logs to Object Storage<a class="zola-anchor" href="#shipping-access-logs-to-object-storage" aria-label="Anchor link for: shipping-access-logs-to-object-storage">🔗</a></h2> <p>Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches.</p> <p><img src="/img/logflow.drawio.png" alt="AWFFull Log Flow — Client → Web Server → Shipper → Object Storage" /></p> <link rel="stylesheet" href="https://blog.none.at/admonition.css?h=98c1477488c4b2c9d71a" type="text/css"> <div class="my-4 flex flex-col rounded-lg bg-(--admonition-bg)" style="--admonition-bg: var(--admonition-danger);" > <div class="flex items-center rounded-t-lg bg-(--admonition-bg) p-1"> <div class="mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert" style="--url: url(icons/danger.svg);" > danger </div> <span><strong>Critical requirement for AWFFull compatibility</strong></span> </div> <div class="pl-4"><p>The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON.</p> </div> </div> <hr /> <h3 id="fluent-bit-s3">Fluent Bit → S3<a class="zola-anchor" href="#fluent-bit-s3" aria-label="Anchor link for: fluent-bit-s3">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://fluentbit.io/">Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins.</p> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/fluent-bit/fluent-bit.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">INPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name tail</span></span> <span class="giallo-l"><span> Path /var/log/nginx/access.log</span></span> <span class="giallo-l"><span> Tag nginx.access</span></span> <span class="giallo-l"><span> Read_from_Head false</span></span> <span class="giallo-l"><span> Refresh_Interval 5</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name s3</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> bucket my-logs</span></span> <span class="giallo-l"><span> region eu-central-1</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Date-partitioned path — one directory per day.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> $UUID is a Fluent Bit built-in variable; it ensures unique filenames</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> when multiple instances run concurrently (e.g. multiple Pods).</span></span> <span class="giallo-l"><span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> <span class="giallo-l"><span> compression gzip</span></span> <span class="giallo-l"><span> total_file_size 50M </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> flush when file reaches 50 MB</span></span> <span class="giallo-l"><span> upload_timeout 10m </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> or after 10 minutes, whichever comes first</span></span> <span class="giallo-l"><span> use_put_object On</span></span></code></pre> <p>For <strong>Azure Blob</strong> replace the output section:</p> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name azure_blob</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> account_name mystorageaccount</span></span> <span class="giallo-l"><span> shared_key ${AZURE_STORAGE_KEY}</span></span> <span class="giallo-l"><span> container_name nginx-logs</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> The azure_blob plugin does not support $UUID in the path parameter.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Uniqueness is guaranteed because Fluent Bit appends an internal</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> timestamp + sequence counter to each blob name automatically.</span></span> <span class="giallo-l"><span> path nginx/%Y/%m/%d</span></span> <span class="giallo-l"><span> auto_create_container On</span></span> <span class="giallo-l"><span> tls On</span></span> <span class="giallo-l"><span> compress gzip</span></span></code></pre> <hr /> <h3 id="vector-s3-azure-blob">Vector → S3 / Azure Blob<a class="zola-anchor" href="#vector-s3-azure-blob" aria-label="Anchor link for: vector-s3-azure-blob">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://vector.dev/">Vector</a> offers more flexibility for transformations and supports both sinks natively:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> vector.yaml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ources</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ginx_access</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ile</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nclude</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ead_from</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">nd</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> only new lines; use &#39;beginning&#39; for backfill</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">inks</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">3_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">ws_s3</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ucket</span><span class="z-punctuation z-separator">:</span><span class="z-string"> m</span><span class="z-string">y-logs</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">egion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">u-central-1</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Vector automatically appends a UUID to key_prefix — no explicit $UUID needed.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Multiple instances writing concurrently will never collide.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> k</span><span class="z-entity z-name z-tag">ey_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> raw lines — no JSON wrapping</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">atch</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 600</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or 10 minutes</span></span></code></pre> <p>For Azure Blob, replace the sink:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">zure_blob_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">zure_blob</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onnection_string</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">${AZURE_STORAGE_CONNECTION_STRING}</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ontainer_name</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-logs</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Same as S3: Vector appends a UUID automatically — uniqueness is built-in.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">lob_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span></code></pre> <hr /> <h3 id="filebeat-logstash-s3">Filebeat → Logstash → S3<a class="zola-anchor" href="#filebeat-logstash-s3" aria-label="Anchor link for: filebeat-logstash-s3">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://www.elastic.co/docs/reference/beats/filebeat/">Filebeat</a> is the standard shipper in the Elastic Stack. It has an <a rel="noopener external" target="_blank" href="https://www.elastic.co/docs/reference/beats/filebeat/filebeat-modules">nginx module</a> that parses, enriches and structures the access log — but <strong>do not use the nginx module for AWFFull</strong>. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format.</p> <p>Instead, use the generic <code>filestream</code> input with no parsing:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/filebeat/filebeat.yml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.inputs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ilestream</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">d</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-access</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">aths</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No parsers, no module — raw lines only</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">arsers</span><span class="z-punctuation z-separator">:</span><span> [</span><span>]</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">o</span><span class="z-entity z-name z-tag">utput.logstash</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> h</span><span class="z-entity z-name z-tag">osts</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">logstash:5044</span><span class="z-string">&quot;</span><span>]</span></span></code></pre> <p>Filebeat has no native S3 output. Use <strong>Logstash</strong> as the intermediary with the <code>logstash-output-s3</code> plugin:</p> <pre class="giallo z-code"><code data-lang="ruby"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/logstash/conf.d/nginx-s3.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>input </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> beats </span><span class="z-punctuation z-section">{</span><span> port </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 5044</span><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>filter </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No grok, no parsing — pass the raw message through unchanged</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>output </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> s3 </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> bucket </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">my-logs</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> region </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">eu-central-1</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> prefix </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">nginx/%{+YYYY}/%{+MM}/%{+dd}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> codec </span><span class="z-punctuation z-separator">=&gt;</span><span> line </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> raw lines, one per file</span></span> <span class="giallo-l"><span> encoding </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">gzip</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> size_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB per file</span></span> <span class="giallo-l"><span> time_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or rotate after 10 minutes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Credentials from IAM role / environment</span></span> <span class="giallo-l"><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span></code></pre> <p>The <code>codec =&gt; line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default <code>json_lines</code> codec would break compatibility.</p> <p><strong>Note:</strong> Filebeat also supports shipping to Azure via Logstash with the <a rel="noopener external" target="_blank" href="https://github.com/janmg/logstash-output-azure_blob_storage">logstash-output-azure_blob_storage</a> community plugin, using the same pattern.</p> <hr /> <h3 id="concurrent-write-safety">Concurrent Write Safety<a class="zola-anchor" href="#concurrent-write-safety" aria-label="Anchor link for: concurrent-write-safety">🔗</a></h3> <p>When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: <strong>can this corrupt the data?</strong></p> <p><strong>Short answer: no — but data loss is possible if you don’t use unique keys.</strong></p> <p>S3 and Azure Blob both guarantee <strong>atomic PUT semantics</strong> at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible.</p> <p>The real risk is <strong>silent overwrite</strong>: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost.</p> <p>Here is how the tools described above handle this:</p> <table><thead><tr><th>Tool</th><th>Mechanism</th><th>Multiple instances</th></tr></thead><tbody> <tr><td>Fluent Bit → S3</td><td><code>$UUID</code> in <code>s3_key_format</code></td><td>Safe — each instance generates a unique key</td></tr> <tr><td>Vector → S3</td><td>UUID appended automatically to <code>key_prefix</code></td><td>Safe — built-in</td></tr> <tr><td>Vector → Azure Blob</td><td>UUID appended automatically to <code>blob_prefix</code></td><td>Safe — built-in</td></tr> <tr><td>Logstash → S3</td><td><code>hostname + timestamp</code> in the generated filename</td><td>Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> <tr><td>Fluent Bit → Azure Blob (append blob)</td><td>Append blob type with atomic server-side appends</td><td>Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> <tr><td>Fluent Bit → Azure Blob (block blob)</td><td>Path parameter sets the blob name</td><td>Unsafe if multiple instances use the same <code>path</code> without a unique suffix</td></tr> </tbody></table> <p><strong>Practical rule:</strong> always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key.</p> <hr /> <h3 id="log-loss-at-high-traffic-sites">Log Loss at High-Traffic Sites<a class="zola-anchor" href="#log-loss-at-high-traffic-sites" aria-label="Anchor link for: log-loss-at-high-traffic-sites">🔗</a></h3> <p>Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are:</p> <p><strong>1. Memory buffer overflow</strong> Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either <strong>drops new lines silently</strong> or applies <strong>backpressure</strong> upstream — which can slow down the web server itself.</p> <p><strong>2. Network outage or S3/Azure throttling</strong> Object storage APIs return <code>503 Slow Down</code> (S3) or <code>429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload.</p> <p><strong>3. Shipper crash without offset tracking</strong> If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage).</p> <p><strong>4. Log rotation race</strong> <code>logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost.</p> <p>Each tool has a different answer to these problems:</p> <table><thead><tr><th>Tool</th><th>Persistence</th><th>At-least-once guarantee</th></tr></thead><tbody> <tr><td>Fluent Bit</td><td>Filesystem buffer (<code>storage.type filesystem</code>)</td><td>Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> <tr><td>Vector</td><td>Disk buffer (<code>type: disk</code> in buffer config)</td><td>Yes — survives restarts; default is in-memory only</td></tr> <tr><td>Logstash</td><td>Persistent queue (<code>queue.type: persisted</code>)</td><td>Yes — survives restarts; default is in-memory only</td></tr> <tr><td>Filebeat</td><td>Registry file tracks read offset per file</td><td>Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> <p>Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated <strong>and deleted</strong> before Filebeat reads to the end, the unread bytes are lost.</p> <p><strong>Recommendations for high-traffic sites:</strong></p> <ul> <li>Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> <li>Set a generous retry policy and back-off interval for S3/Azure upload errors</li> <li>Configure log rotation with a <code>delaycompress</code> / <code>rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> <li>Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> <p>Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool.</p> <h4 id="fluent-bit-filesystem-buffer-retry">Fluent Bit — Filesystem Buffer + Retry<a class="zola-anchor" href="#fluent-bit-filesystem-buffer-retry" aria-label="Anchor link for: fluent-bit-filesystem-buffer-retry">🔗</a></h4> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/fluent-bit/fluent-bit.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">SERVICE</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Flush 5</span></span> <span class="giallo-l"><span> Daemon Off</span></span> <span class="giallo-l"><span> Log_Level info</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Filesystem buffering — chunks are written to disk before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Data survives Fluent Bit restarts and network outages up to the limit.</span></span> <span class="giallo-l"><span> storage.path /var/lib/fluent-bit/buf/</span></span> <span class="giallo-l"><span> storage.sync normal </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> fsync after each chunk write</span></span> <span class="giallo-l"><span> storage.checksum off</span></span> <span class="giallo-l"><span> storage.max_chunks_up 128 </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> max chunks in memory at once</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">INPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name tail</span></span> <span class="giallo-l"><span> Path /var/log/nginx/access.log</span></span> <span class="giallo-l"><span> Tag nginx.access</span></span> <span class="giallo-l"><span> Read_from_Head false</span></span> <span class="giallo-l"><span> Refresh_Interval 5</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Offset database — Fluent Bit remembers the last read position.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Without this, every restart re-reads the file from the beginning.</span></span> <span class="giallo-l"><span> DB /var/lib/fluent-bit/nginx-access.db</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Use filesystem storage for this input&#39;s chunks (not memory)</span></span> <span class="giallo-l"><span> storage.type filesystem</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name s3</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> bucket my-logs</span></span> <span class="giallo-l"><span> region eu-central-1</span></span> <span class="giallo-l"><span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> <span class="giallo-l"><span> compression gzip</span></span> <span class="giallo-l"><span> total_file_size 50M</span></span> <span class="giallo-l"><span> upload_timeout 10m</span></span> <span class="giallo-l"><span> use_put_object On</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry: up to 5 attempts with exponential back-off (max ~160 s between tries).</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Set False for unlimited retries (use with caution — can block the pipeline).</span></span> <span class="giallo-l"><span> Retry_Limit 5</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Cap the on-disk buffer for this output at 1 GB.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> When full, oldest chunks are dropped to make room.</span></span> <span class="giallo-l"><span> storage.total_limit_size 1G</span></span></code></pre> <p>The <code>DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if <code>logrotate</code> renames <code>access.log</code> to <code>access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file.</p> <hr /> <h4 id="vector-disk-buffer-retry">Vector — Disk Buffer + Retry<a class="zola-anchor" href="#vector-disk-buffer-retry" aria-label="Anchor link for: vector-disk-buffer-retry">🔗</a></h4> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/vector/vector.yaml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ources</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ginx_access</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ile</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nclude</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ead_from</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">nd</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Vector also writes a checkpoint file (similar to Filebeat&#39;s registry)</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> to /var/lib/vector/ by default — survives restarts automatically.</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">inks</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">3_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">ws_s3</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ucket</span><span class="z-punctuation z-separator">:</span><span class="z-string"> m</span><span class="z-string">y-logs</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">egion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">u-central-1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> k</span><span class="z-entity z-name z-tag">ey_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">atch</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB per object</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 600</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or 10 minutes, whichever comes first</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Persistent disk buffer — events are written to disk before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Survives Vector restarts; default is type: memory (lost on crash).</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">uffer</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> d</span><span class="z-string">isk</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_size</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1073741824</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 1 GB</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> w</span><span class="z-entity z-name z-tag">hen_full</span><span class="z-punctuation z-separator">:</span><span class="z-string"> b</span><span class="z-string">lock</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> apply backpressure instead of dropping events</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry policy for S3 upload failures (503, 429, network errors).</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">equest</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_attempts</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> max upload attempts per batch</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_initial_backoff_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> first retry after 1 s</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_max_duration_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 300</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> cap back-off at 5 minutes</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 60</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> per-request timeout</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ate_limit_num</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 100</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> max concurrent requests</span></span></code></pre> <p><code>when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use <code>when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog.</p> <hr /> <h4 id="filebeat-logstash-registry-persistent-queue">Filebeat + Logstash — Registry + Persistent Queue<a class="zola-anchor" href="#filebeat-logstash-registry-persistent-queue" aria-label="Anchor link for: filebeat-logstash-registry-persistent-queue">🔗</a></h4> <p><strong>Filebeat</strong> (<code>/etc/filebeat/filebeat.yml</code>):</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.inputs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ilestream</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">d</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-access</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">aths</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">arsers</span><span class="z-punctuation z-separator">:</span><span> [</span><span>]</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> no parsing — raw lines only</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Filebeat writes a registry to /var/lib/filebeat/registry/ that records</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> the inode, byte offset, and identifier for every tracked file.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> On restart it resumes exactly where it stopped — no duplicates, no gaps.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.registry.path</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/filebeat/registry</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Internal memory queue — tune for throughput vs. latency.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> These events are lost on crash; durability comes from the registry,</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> not from this queue.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.mem</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">vents</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 4096</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> f</span><span class="z-entity z-name z-tag">lush.min_events</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 512</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> f</span><span class="z-entity z-name z-tag">lush.timeout</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 5</span><span class="z-string">s</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">o</span><span class="z-entity z-name z-tag">utput.logstash</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> h</span><span class="z-entity z-name z-tag">osts</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">logstash:5044</span><span class="z-string">&quot;</span><span>]</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry connecting to Logstash — Filebeat will keep trying indefinitely.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Events accumulate in the queue while Logstash is unreachable.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 30</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_retries</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 3</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> per batch; 0 = retry forever</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ulk_max_size</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 2048</span></span></code></pre> <p><strong>Logstash</strong> (<code>/etc/logstash/logstash.yml</code>):</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Persistent queue — events are written to disk between the input and filter/</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> output stages. Logstash survives restarts without losing in-flight events.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.type</span><span class="z-punctuation z-separator">:</span><span class="z-string"> p</span><span class="z-string">ersisted</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.max_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 1</span><span class="z-string">gb</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.path</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/logstash/queue</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Checkpoint frequency — lower values = more durable, more I/O.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.checkpoint.acks</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1024</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.checkpoint.writes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1024</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Dead-letter queue — events that fail after all retries go here instead</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> of being silently dropped. Inspect with the dead_letter_queue input plugin.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">d</span><span class="z-entity z-name z-tag">ead_letter_queue.enable</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">d</span><span class="z-entity z-name z-tag">ead_letter_queue.max_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 1</span><span class="z-string">gb</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">p</span><span class="z-entity z-name z-tag">ath.dead_letter_queue</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/logstash/dead_letter_queue</span></span></code></pre> <p><strong>Logstash pipeline</strong> (<code>/etc/logstash/conf.d/nginx-s3.conf</code>):</p> <pre class="giallo z-code"><code data-lang="ruby"><span class="giallo-l"><span>input </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> beats </span><span class="z-punctuation z-section">{</span><span> port </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 5044</span><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>filter </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No grok, no parsing — pass the raw message through unchanged</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>output </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> s3 </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> bucket </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">my-logs</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> region </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">eu-central-1</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> prefix </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">nginx/%{+YYYY}/%{+MM}/%{+dd}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> codec </span><span class="z-punctuation z-separator">=&gt;</span><span> line </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> one raw log line per line</span></span> <span class="giallo-l"><span> encoding </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">gzip</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> size_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> rotate object at 50 MB</span></span> <span class="giallo-l"><span> time_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or after 10 minutes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Local staging directory — files are assembled here before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Use a path on a real disk, not tmpfs, for crash safety.</span></span> <span class="giallo-l"><span> temporary_directory </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">/var/lib/logstash/s3-tmp</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry failed uploads before giving up and sending to the DLQ.</span></span> <span class="giallo-l"><span> retry_count </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span></span> <span class="giallo-l"><span> retry_delay </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> seconds between retries</span></span> <span class="giallo-l"><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span></code></pre> <p>With <code>queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped.</p> <hr /> <h2 id="running-awffull-against-object-storage-logs">Running AWFFull Against Object Storage Logs<a class="zola-anchor" href="#running-awffull-against-object-storage-logs" aria-label="Anchor link for: running-awffull-against-object-storage-logs">🔗</a></h2> <p><img src="/img/reportflow.drawio.png" alt="AWFFull Report Pipeline — Object Storage → AWFFull → Object Storage" /></p> <div class="my-4 flex flex-col rounded-lg bg-(--admonition-bg)" style="--admonition-bg: var(--admonition-info);" > <div class="flex items-center rounded-t-lg bg-(--admonition-bg) p-1"> <div class="mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert" style="--url: url(icons/info.svg);" > info </div> <span><strong>Info</strong></span> </div> <div class="pl-4"><p>The AWFFull container downloads the log files and the previous <code>awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed.</p> </div> </div> <p>Once logs are stored with date-based prefixes, fetching a specific day is a single command:</p> <p><strong>From S3:</strong></p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Download all log files for 12 March 2026</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/12/</span><span class="z-string"> /tmp/logs/2026-03-12/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Process with AWFFull (pass files in chronological order)</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">ls</span><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-variable z-language">*</span><span class="z-string">.log.gz</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p><strong>From Azure Blob:</strong></p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> &quot;</span><span class="z-string">https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/*</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">ls</span><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-variable z-language">*</span><span class="z-string">.log.gz</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p><strong>Downloading a full month</strong> (common for monthly reporting):</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/</span><span class="z-string"> /tmp/logs/2026-03/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.gz</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">find</span><span class="z-string"> /tmp/logs/2026-03/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">name</span><span class="z-string"> &quot;</span><span class="z-string">*.log.gz</span><span class="z-string">&quot;</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p>The <code>sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. <code>access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically.</p> <hr /> <h2 id="running-awffull-in-a-container-world">Running AWFFull in a Container World<a class="zola-anchor" href="#running-awffull-in-a-container-world" aria-label="Anchor link for: running-awffull-in-a-container-world">🔗</a></h2> <p>AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns:</p> <hr /> <h3 id="option-1-persistent-volume-classical-approach">Option 1 — Persistent Volume (classical approach)<a class="zola-anchor" href="#option-1-persistent-volume-classical-approach" aria-label="Anchor link for: option-1-persistent-volume-classical-approach">🔗</a></h3> <p>The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup.</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> docker-compose example</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ervices</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">wffull</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">mage</span><span class="z-punctuation z-separator">:</span><span class="z-string"> r</span><span class="z-string">egistry.gitlab.com/aleks001/awfull:latest</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumes</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx:/logs:ro</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> log files (read-only)</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">srv/awffull/reports:/reports</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> generated HTML + PNG output</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">etc/awffull.conf:/etc/awffull.conf:ro</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ommand</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">-c</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/etc/awffull.conf</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/logs/access.log.gz</span><span class="z-string">&quot;</span><span>]</span></span></code></pre> <p>In Kubernetes, replace the host-path mounts with a <code>PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here.</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Kubernetes CronJob skeleton</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">a</span><span class="z-entity z-name z-tag">piVersion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> b</span><span class="z-string">atch/v1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">k</span><span class="z-entity z-name z-tag">ind</span><span class="z-punctuation z-separator">:</span><span class="z-string"> C</span><span class="z-string">ronJob</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">m</span><span class="z-entity z-name z-tag">etadata</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-monthly</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">chedule</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">0 2 1 * *</span><span class="z-string">&quot;</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 02:00 on the 1st of each month</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> j</span><span class="z-entity z-name z-tag">obTemplate</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">emplate</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ontainers</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">mage</span><span class="z-punctuation z-separator">:</span><span class="z-string"> r</span><span class="z-string">egistry.gitlab.com/aleks001/awfull:latest</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">rgs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">-c</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/etc/awffull.conf</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/logs/access.log.gz</span><span class="z-string">&quot;</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumeMounts</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> l</span><span class="z-entity z-name z-tag">ogs mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">logs</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">eports mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">reports</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onfig mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">etc/awffull.conf</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">ubPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull.conf</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumes</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> l</span><span class="z-entity z-name z-tag">ogs persistentVolumeClaim</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">laimName</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-logs</span><span> }</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">eports persistentVolumeClaim</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">laimName</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-reports</span><span> }</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onfig configMap</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-config</span><span> }</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">estartPolicy</span><span class="z-punctuation z-separator">:</span><span class="z-string"> O</span><span class="z-string">nFailure</span></span></code></pre> <p><strong>Trade-off:</strong> Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod).</p> <hr /> <h3 id="option-2-s3-s3-compatible-object-storage">Option 2 — S3 / S3-compatible Object Storage<a class="zola-anchor" href="#option-2-s3-s3-compatible-object-storage" aria-label="Anchor link for: option-2-s3-s3-compatible-object-storage">🔗</a></h3> <p>This option covers two related questions:</p> <ol> <li>Where do the generated reports go?</li> <li><strong>Can logs that were already shipped to S3 be fed back into AWFFull?</strong></li> </ol> <p>The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline.</p> <p><strong>Shipping logs to S3</strong> is standard practice. Tools like <a rel="noopener external" target="_blank" href="https://fluentbit.io/">Fluent Bit</a>, <a rel="noopener external" target="_blank" href="https://vector.dev/">Vector</a>, or even a plain <code>logrotate</code> + <code>aws s3 cp</code> cron job send compressed log files to a bucket with a path like <code>s3://my-logs/nginx/2026/03/01/access.log.gz</code>.</p> <p><strong>Reading them back</strong> for AWFFull is equally straightforward:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1. Download history file (needed for incremental processing)</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> cp</span><span class="z-string"> s3://my-bucket/awffull/awffull.hist</span><span class="z-string"> /var/lib/awffull/</span><span class="z-keyword z-keyword z-operator"> 2</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-string">/dev/null</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-keyword z-keyword z-operator">|</span><span class="z-support z-support z-function"> true</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2. Download log files for the period to process</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/</span><span class="z-string"> /tmp/logs/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.gz</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3. Run AWFFull (reads logs, updates history, writes HTML)</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /tmp/logs/</span><span class="z-variable z-language">*</span><span class="z-string">.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4. Persist the updated history file back to S3</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> cp</span><span class="z-string"> /var/lib/awffull/awffull.hist</span><span class="z-string"> s3://my-bucket/awffull/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 5. Sync reports to the reports bucket / static website</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-reports-bucket/awffull/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete</span></span></code></pre> <p>The <code>awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (<code>-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well.</p> <p>In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist)</span></span> <span class="giallo-l"><span> ▲</span></span> <span class="giallo-l"><span> └── IAM role / workload identity</span></span></code></pre> <p>Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.).</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> entrypoint wrapper</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /logs/access.log.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Sync output to S3</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-bucket/awffull/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-content-type</span><span class="z-string"> &quot;</span><span class="z-string">text/html</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-exclude</span><span class="z-string"> &quot;</span><span class="z-string">*.png</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.html</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-bucket/awffull/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-exclude</span><span class="z-string"> &quot;</span><span class="z-string">*.html</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.png</span><span class="z-string">&quot;</span></span></code></pre> <p>In a container, add <code>awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables.</p> <p><strong>Trade-off:</strong> Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions.</p> <hr /> <h3 id="option-3-azure-blob-storage">Option 3 — Azure Blob Storage<a class="zola-anchor" href="#option-3-azure-blob-storage" aria-label="Anchor link for: option-3-azure-blob-storage">🔗</a></h3> <p>Same pattern as S3, using the Azure CLI (<code>az storage blob</code>) or <code>azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint.</p> <p>Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-variable z-other z-variable">STORAGE</span><span class="z-keyword z-keyword z-operator">=</span><span class="z-string">&quot;</span><span class="z-string">https://&lt;storage-account&gt;.blob.core.windows.net</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1. Download history file</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/awffull/awffull.hist</span><span class="z-string">&quot;</span><span class="z-string"> /var/lib/awffull/</span><span class="z-keyword z-keyword z-operator"> 2</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-string">/dev/null</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-keyword z-keyword z-operator">|</span><span class="z-support z-support z-function"> true</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2. Download log files for the period</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/nginx-logs/2026/03/*</span><span class="z-string">&quot;</span><span class="z-string"> /tmp/logs/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3. Run AWFFull</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /tmp/logs/</span><span class="z-variable z-language">*</span><span class="z-string">.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4. Persist history back to Blob</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> /var/lib/awffull/awffull.hist</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/awffull/awffull.hist</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 5. Sync reports to static website container ($web)</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/</span><span class="z-constant z-character">\$</span><span class="z-string">web/awffull/</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete-destination=true</span></span></code></pre> <p>Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> With workload identity (AKS)</span></span> <span class="giallo-l"><span class="z-entity z-name">az</span><span class="z-string"> login</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-identity</span></span> <span class="giallo-l"><span class="z-entity z-name">az</span><span class="z-string"> storage</span><span class="z-string"> blob</span><span class="z-string"> sync</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-source</span><span class="z-string"> /reports/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-container</span><span class="z-string"> &#39;</span><span class="z-string">$web</span><span class="z-string">&#39;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-account-name</span><span class="z-keyword z-keyword z-operator"> &lt;</span><span class="z-string">storage-accoun</span><span>t</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-destination</span><span class="z-string"> awffull/</span></span></code></pre> <p>Enable the <strong>Static website</strong> feature on the Storage Account and the reports are immediately reachable at <code>https://&lt;storage-account&gt;.z6.web.core.windows.net/awffull/index.html</code>.</p> <p><strong>Trade-off:</strong> Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials.</p> <hr /> <h3 id="choosing-a-pattern">Choosing a Pattern<a class="zola-anchor" href="#choosing-a-pattern" aria-label="Anchor link for: choosing-a-pattern">🔗</a></h3> <table><thead><tr><th>Scenario</th><th>Recommended approach</th></tr></thead><tbody> <tr><td>Simple on-prem Kubernetes</td><td>Persistent Volume + nginx sidecar</td></tr> <tr><td>AWS / EKS</td><td>S3 + CloudFront static website</td></tr> <tr><td>Azure / AKS</td><td>Azure Blob static website + workload identity</td></tr> <tr><td>Self-hosted, no cloud</td><td>S3 sync</td></tr> <tr><td>CI/CD pipeline artifact</td><td>Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> <p>In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (<code>awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path.</p> <hr /> <p><em>AWFFull is free software licensed under GPL v3 or later.</em> <em>The project is at <a rel="noopener external" target="_blank" href="https://gitlab.com/aleks001/awfful">https://gitlab.com/aleks001/awfful</a>.</em></p>