https://blog.none.at Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola en Sat, 27 Jun 2026 00:00:00 +0000 Mon, 18 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-18-signoz-on-ovh-usecase-access-logs/ https://blog.none.at/blog/2026/2026-05-18-signoz-on-ovh-usecase-access-logs/ <p>Posts <a href="https://blog.none.at/blog/2026/2026-05-16-signoz-on-ovh-infrastructure/">1</a> and <a href="https://blog.none.at/blog/2026/2026-05-16-signoz-on-ovh-metrics-traces-logs/">2</a> of this series set up the SigNoz observability stack on OVH MKS and demonstrated metrics, traces, and Kubernetes pod log collection. This post adds a practical use case: structured Envoy access log collection via Vector, archiving to OVH Object Storage in Combined Log Format, and monthly HTML reports via awffull — accessible at <code>https://reports.&lt;your-domain&gt;</code> through the existing Istio gateway.</p> Thu, 14 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-operations/ https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-operations/ <p>This post covers the operational dimension of the <a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving/">managed log archiving comparison</a>: how each provider integrates with Kubernetes, what ingest pipeline options exist, how backup and disaster recovery work, and how you observe the log pipeline itself.</p> <ul> <li><strong>Full guide:</strong> <a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-guide/">AWS vs. GCP vs. Azure vs. OVHcloud: The Complete Guide to Managed Log Archiving</a></li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving/">Part 1 — Overview &amp; Cost:</a> Services, storage model, 7-year cost comparison</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-preflight/">Part 2 — Pre-Flight, Flexibility &amp; Auditor Export:</a> Pre-flight checklist, retroactive flexibility, auditor export</li> <li><strong>Part 3 — Operations — this post:</strong> Kubernetes integration, ingest pipeline, backup &amp; DR, observability</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-security/">Part 4 — Security &amp; Compliance:</a> IAM/RBAC, encryption, WORM, GDPR, compliance certificates</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-ux/">Part 5 — Query, Dashboards &amp; Recommendations:</a> Query interfaces, dashboards, alerting, cold-tier behaviour, when to use which</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-aws-gcp-azure-ovh-log-archiving-production/">Part 6 — Production Checklist, Guardrails &amp; Runbooks:</a> Loss detection, privacy, schema, cost guardrails, runbooks, fire drills</li> </ul> <p><em>Self-hosted alternative: <a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-operations/">Elasticsearch vs. OpenSearch vs. Loki vs. Quickwit vs. ClickHouse — Part 2 (Operations)</a></em></p> Thu, 14 May 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-usecases/ https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-usecases/ <p>The first four parts of this series compared Elasticsearch, OpenSearch, Loki, Quickwit, and ClickHouse across storage models, operations, security, and UX. This part shows what a concrete production setup looks like when the decision has been made: <strong>ClickHouse</strong> as the log store, <strong>Vector</strong> as the ingestion agent, running on <strong>OVH Managed Kubernetes Service (MKS)</strong> with <strong>OVH Object Storage</strong> as the cold tier.</p> <p>This is Part 5 of a five-part series:</p> <ul> <li><strong>Full guide:</strong> <a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-guide/">Elasticsearch vs. OpenSearch vs. Loki vs. Quickwit vs. ClickHouse: The Complete Guide</a></li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse/">Part 1</a> — storage models, compression, resource consumption, and SaaS options</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-operations/">Part 2</a> — cluster setup, Kubernetes operations, backup and disaster recovery</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-security/">Part 3</a> — encryption, RBAC, and WORM compliance</li> <li><a href="https://blog.none.at/blog/2026/2026-05-14-es-os-loki-quickwit-clickhouse-ux/">Part 4</a> — UI layers, dashboards, alerting, and cold-tier query behaviour</li> <li>Part 5 — this post</li> </ul> Thu, 12 Mar 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-03-12-blog-log-shipping/ https://blog.none.at/blog/2026/2026-03-12-blog-log-shipping/ <p>This is a companion post to <a href="https://blog.none.at/blog/2026/2026-03-12-blog-awffull/">AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs.</p> <hr /> <h2 id="shipping-access-logs-to-object-storage">Shipping Access Logs to Object Storage<a class="zola-anchor" href="#shipping-access-logs-to-object-storage" aria-label="Anchor link for: shipping-access-logs-to-object-storage">🔗</a></h2> <p>Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches.</p> <p><img src="/img/logflow.drawio.png" alt="AWFFull Log Flow — Client → Web Server → Shipper → Object Storage" /></p> <link rel="stylesheet" href="https://blog.none.at/admonition.css?h=98c1477488c4b2c9d71a" type="text/css"> <div class="my-4 flex flex-col rounded-lg bg-(--admonition-bg)" style="--admonition-bg: var(--admonition-danger);" > <div class="flex items-center rounded-t-lg bg-(--admonition-bg) p-1"> <div class="mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert" style="--url: url(icons/danger.svg);" > danger </div> <span><strong>Critical requirement for AWFFull compatibility</strong></span> </div> <div class="pl-4"><p>The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON.</p> </div> </div> <hr /> <h3 id="fluent-bit-s3">Fluent Bit → S3<a class="zola-anchor" href="#fluent-bit-s3" aria-label="Anchor link for: fluent-bit-s3">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://fluentbit.io/">Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins.</p> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/fluent-bit/fluent-bit.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">INPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name tail</span></span> <span class="giallo-l"><span> Path /var/log/nginx/access.log</span></span> <span class="giallo-l"><span> Tag nginx.access</span></span> <span class="giallo-l"><span> Read_from_Head false</span></span> <span class="giallo-l"><span> Refresh_Interval 5</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name s3</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> bucket my-logs</span></span> <span class="giallo-l"><span> region eu-central-1</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Date-partitioned path — one directory per day.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> $UUID is a Fluent Bit built-in variable; it ensures unique filenames</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> when multiple instances run concurrently (e.g. multiple Pods).</span></span> <span class="giallo-l"><span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> <span class="giallo-l"><span> compression gzip</span></span> <span class="giallo-l"><span> total_file_size 50M </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> flush when file reaches 50 MB</span></span> <span class="giallo-l"><span> upload_timeout 10m </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> or after 10 minutes, whichever comes first</span></span> <span class="giallo-l"><span> use_put_object On</span></span></code></pre> <p>For <strong>Azure Blob</strong> replace the output section:</p> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name azure_blob</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> account_name mystorageaccount</span></span> <span class="giallo-l"><span> shared_key ${AZURE_STORAGE_KEY}</span></span> <span class="giallo-l"><span> container_name nginx-logs</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> The azure_blob plugin does not support $UUID in the path parameter.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Uniqueness is guaranteed because Fluent Bit appends an internal</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> timestamp + sequence counter to each blob name automatically.</span></span> <span class="giallo-l"><span> path nginx/%Y/%m/%d</span></span> <span class="giallo-l"><span> auto_create_container On</span></span> <span class="giallo-l"><span> tls On</span></span> <span class="giallo-l"><span> compress gzip</span></span></code></pre> <hr /> <h3 id="vector-s3-azure-blob">Vector → S3 / Azure Blob<a class="zola-anchor" href="#vector-s3-azure-blob" aria-label="Anchor link for: vector-s3-azure-blob">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://vector.dev/">Vector</a> offers more flexibility for transformations and supports both sinks natively:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> vector.yaml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ources</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ginx_access</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ile</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nclude</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ead_from</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">nd</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> only new lines; use &#39;beginning&#39; for backfill</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">inks</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">3_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">ws_s3</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ucket</span><span class="z-punctuation z-separator">:</span><span class="z-string"> m</span><span class="z-string">y-logs</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">egion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">u-central-1</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Vector automatically appends a UUID to key_prefix — no explicit $UUID needed.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Multiple instances writing concurrently will never collide.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> k</span><span class="z-entity z-name z-tag">ey_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> raw lines — no JSON wrapping</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">atch</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 600</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or 10 minutes</span></span></code></pre> <p>For Azure Blob, replace the sink:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">zure_blob_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">zure_blob</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onnection_string</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">${AZURE_STORAGE_CONNECTION_STRING}</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ontainer_name</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-logs</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Same as S3: Vector appends a UUID automatically — uniqueness is built-in.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">lob_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span></code></pre> <hr /> <h3 id="filebeat-logstash-s3">Filebeat → Logstash → S3<a class="zola-anchor" href="#filebeat-logstash-s3" aria-label="Anchor link for: filebeat-logstash-s3">🔗</a></h3> <p><a rel="noopener external" target="_blank" href="https://www.elastic.co/docs/reference/beats/filebeat/">Filebeat</a> is the standard shipper in the Elastic Stack. It has an <a rel="noopener external" target="_blank" href="https://www.elastic.co/docs/reference/beats/filebeat/filebeat-modules">nginx module</a> that parses, enriches and structures the access log — but <strong>do not use the nginx module for AWFFull</strong>. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format.</p> <p>Instead, use the generic <code>filestream</code> input with no parsing:</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/filebeat/filebeat.yml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.inputs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ilestream</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">d</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-access</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">aths</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No parsers, no module — raw lines only</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">arsers</span><span class="z-punctuation z-separator">:</span><span> [</span><span>]</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">o</span><span class="z-entity z-name z-tag">utput.logstash</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> h</span><span class="z-entity z-name z-tag">osts</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">logstash:5044</span><span class="z-string">&quot;</span><span>]</span></span></code></pre> <p>Filebeat has no native S3 output. Use <strong>Logstash</strong> as the intermediary with the <code>logstash-output-s3</code> plugin:</p> <pre class="giallo z-code"><code data-lang="ruby"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/logstash/conf.d/nginx-s3.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>input </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> beats </span><span class="z-punctuation z-section">{</span><span> port </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 5044</span><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>filter </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No grok, no parsing — pass the raw message through unchanged</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>output </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> s3 </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> bucket </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">my-logs</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> region </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">eu-central-1</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> prefix </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">nginx/%{+YYYY}/%{+MM}/%{+dd}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> codec </span><span class="z-punctuation z-separator">=&gt;</span><span> line </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> raw lines, one per file</span></span> <span class="giallo-l"><span> encoding </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">gzip</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> size_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB per file</span></span> <span class="giallo-l"><span> time_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or rotate after 10 minutes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Credentials from IAM role / environment</span></span> <span class="giallo-l"><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span></code></pre> <p>The <code>codec =&gt; line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default <code>json_lines</code> codec would break compatibility.</p> <p><strong>Note:</strong> Filebeat also supports shipping to Azure via Logstash with the <a rel="noopener external" target="_blank" href="https://github.com/janmg/logstash-output-azure_blob_storage">logstash-output-azure_blob_storage</a> community plugin, using the same pattern.</p> <hr /> <h3 id="concurrent-write-safety">Concurrent Write Safety<a class="zola-anchor" href="#concurrent-write-safety" aria-label="Anchor link for: concurrent-write-safety">🔗</a></h3> <p>When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: <strong>can this corrupt the data?</strong></p> <p><strong>Short answer: no — but data loss is possible if you don’t use unique keys.</strong></p> <p>S3 and Azure Blob both guarantee <strong>atomic PUT semantics</strong> at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible.</p> <p>The real risk is <strong>silent overwrite</strong>: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost.</p> <p>Here is how the tools described above handle this:</p> <table><thead><tr><th>Tool</th><th>Mechanism</th><th>Multiple instances</th></tr></thead><tbody> <tr><td>Fluent Bit → S3</td><td><code>$UUID</code> in <code>s3_key_format</code></td><td>Safe — each instance generates a unique key</td></tr> <tr><td>Vector → S3</td><td>UUID appended automatically to <code>key_prefix</code></td><td>Safe — built-in</td></tr> <tr><td>Vector → Azure Blob</td><td>UUID appended automatically to <code>blob_prefix</code></td><td>Safe — built-in</td></tr> <tr><td>Logstash → S3</td><td><code>hostname + timestamp</code> in the generated filename</td><td>Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> <tr><td>Fluent Bit → Azure Blob (append blob)</td><td>Append blob type with atomic server-side appends</td><td>Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> <tr><td>Fluent Bit → Azure Blob (block blob)</td><td>Path parameter sets the blob name</td><td>Unsafe if multiple instances use the same <code>path</code> without a unique suffix</td></tr> </tbody></table> <p><strong>Practical rule:</strong> always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key.</p> <hr /> <h3 id="log-loss-at-high-traffic-sites">Log Loss at High-Traffic Sites<a class="zola-anchor" href="#log-loss-at-high-traffic-sites" aria-label="Anchor link for: log-loss-at-high-traffic-sites">🔗</a></h3> <p>Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are:</p> <p><strong>1. Memory buffer overflow</strong> Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either <strong>drops new lines silently</strong> or applies <strong>backpressure</strong> upstream — which can slow down the web server itself.</p> <p><strong>2. Network outage or S3/Azure throttling</strong> Object storage APIs return <code>503 Slow Down</code> (S3) or <code>429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload.</p> <p><strong>3. Shipper crash without offset tracking</strong> If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage).</p> <p><strong>4. Log rotation race</strong> <code>logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost.</p> <p>Each tool has a different answer to these problems:</p> <table><thead><tr><th>Tool</th><th>Persistence</th><th>At-least-once guarantee</th></tr></thead><tbody> <tr><td>Fluent Bit</td><td>Filesystem buffer (<code>storage.type filesystem</code>)</td><td>Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> <tr><td>Vector</td><td>Disk buffer (<code>type: disk</code> in buffer config)</td><td>Yes — survives restarts; default is in-memory only</td></tr> <tr><td>Logstash</td><td>Persistent queue (<code>queue.type: persisted</code>)</td><td>Yes — survives restarts; default is in-memory only</td></tr> <tr><td>Filebeat</td><td>Registry file tracks read offset per file</td><td>Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> <p>Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated <strong>and deleted</strong> before Filebeat reads to the end, the unread bytes are lost.</p> <p><strong>Recommendations for high-traffic sites:</strong></p> <ul> <li>Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> <li>Set a generous retry policy and back-off interval for S3/Azure upload errors</li> <li>Configure log rotation with a <code>delaycompress</code> / <code>rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> <li>Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> <p>Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool.</p> <h4 id="fluent-bit-filesystem-buffer-retry">Fluent Bit — Filesystem Buffer + Retry<a class="zola-anchor" href="#fluent-bit-filesystem-buffer-retry" aria-label="Anchor link for: fluent-bit-filesystem-buffer-retry">🔗</a></h4> <pre class="giallo z-code"><code data-lang="ini"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/fluent-bit/fluent-bit.conf</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">SERVICE</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Flush 5</span></span> <span class="giallo-l"><span> Daemon Off</span></span> <span class="giallo-l"><span> Log_Level info</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Filesystem buffering — chunks are written to disk before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Data survives Fluent Bit restarts and network outages up to the limit.</span></span> <span class="giallo-l"><span> storage.path /var/lib/fluent-bit/buf/</span></span> <span class="giallo-l"><span> storage.sync normal </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> fsync after each chunk write</span></span> <span class="giallo-l"><span> storage.checksum off</span></span> <span class="giallo-l"><span> storage.max_chunks_up 128 </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> max chunks in memory at once</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">INPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name tail</span></span> <span class="giallo-l"><span> Path /var/log/nginx/access.log</span></span> <span class="giallo-l"><span> Tag nginx.access</span></span> <span class="giallo-l"><span> Read_from_Head false</span></span> <span class="giallo-l"><span> Refresh_Interval 5</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Offset database — Fluent Bit remembers the last read position.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Without this, every restart re-reads the file from the beginning.</span></span> <span class="giallo-l"><span> DB /var/lib/fluent-bit/nginx-access.db</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Use filesystem storage for this input&#39;s chunks (not memory)</span></span> <span class="giallo-l"><span> storage.type filesystem</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">[</span><span class="z-entity z-name">OUTPUT</span><span class="z-entity z-name">]</span></span> <span class="giallo-l"><span> Name s3</span></span> <span class="giallo-l"><span> Match nginx.access</span></span> <span class="giallo-l"><span> bucket my-logs</span></span> <span class="giallo-l"><span> region eu-central-1</span></span> <span class="giallo-l"><span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> <span class="giallo-l"><span> compression gzip</span></span> <span class="giallo-l"><span> total_file_size 50M</span></span> <span class="giallo-l"><span> upload_timeout 10m</span></span> <span class="giallo-l"><span> use_put_object On</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry: up to 5 attempts with exponential back-off (max ~160 s between tries).</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Set False for unlimited retries (use with caution — can block the pipeline).</span></span> <span class="giallo-l"><span> Retry_Limit 5</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Cap the on-disk buffer for this output at 1 GB.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> When full, oldest chunks are dropped to make room.</span></span> <span class="giallo-l"><span> storage.total_limit_size 1G</span></span></code></pre> <p>The <code>DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if <code>logrotate</code> renames <code>access.log</code> to <code>access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file.</p> <hr /> <h4 id="vector-disk-buffer-retry">Vector — Disk Buffer + Retry<a class="zola-anchor" href="#vector-disk-buffer-retry" aria-label="Anchor link for: vector-disk-buffer-retry">🔗</a></h4> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> /etc/vector/vector.yaml</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ources</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ginx_access</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ile</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nclude</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ead_from</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">nd</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Vector also writes a checkpoint file (similar to Filebeat&#39;s registry)</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> to /var/lib/vector/ by default — survives restarts automatically.</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">inks</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">3_logs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">ws_s3</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">nputs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">n</span><span class="z-string">ginx_access</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ucket</span><span class="z-punctuation z-separator">:</span><span class="z-string"> m</span><span class="z-string">y-logs</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">egion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> e</span><span class="z-string">u-central-1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> k</span><span class="z-entity z-name z-tag">ey_prefix</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">nginx/{{ now() | strftime(&#39;%Y/%m/%d&#39;) }}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ompression</span><span class="z-punctuation z-separator">:</span><span class="z-string"> g</span><span class="z-string">zip</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">ncoding</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">odec</span><span class="z-punctuation z-separator">:</span><span class="z-string"> t</span><span class="z-string">ext</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">atch</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 50 MB per object</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 600</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or 10 minutes, whichever comes first</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Persistent disk buffer — events are written to disk before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Survives Vector restarts; default is type: memory (lost on crash).</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">uffer</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> d</span><span class="z-string">isk</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_size</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1073741824</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 1 GB</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> w</span><span class="z-entity z-name z-tag">hen_full</span><span class="z-punctuation z-separator">:</span><span class="z-string"> b</span><span class="z-string">lock</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> apply backpressure instead of dropping events</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry policy for S3 upload failures (503, 429, network errors).</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">equest</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_attempts</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> max upload attempts per batch</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_initial_backoff_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> first retry after 1 s</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">etry_max_duration_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 300</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> cap back-off at 5 minutes</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout_secs</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 60</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> per-request timeout</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">ate_limit_num</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 100</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> max concurrent requests</span></span></code></pre> <p><code>when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use <code>when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog.</p> <hr /> <h4 id="filebeat-logstash-registry-persistent-queue">Filebeat + Logstash — Registry + Persistent Queue<a class="zola-anchor" href="#filebeat-logstash-registry-persistent-queue" aria-label="Anchor link for: filebeat-logstash-registry-persistent-queue">🔗</a></h4> <p><strong>Filebeat</strong> (<code>/etc/filebeat/filebeat.yml</code>):</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.inputs</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">ype</span><span class="z-punctuation z-separator">:</span><span class="z-string"> f</span><span class="z-string">ilestream</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">d</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-access</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">aths</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx/access.log</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> p</span><span class="z-entity z-name z-tag">arsers</span><span class="z-punctuation z-separator">:</span><span> [</span><span>]</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> no parsing — raw lines only</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Filebeat writes a registry to /var/lib/filebeat/registry/ that records</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> the inode, byte offset, and identifier for every tracked file.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> On restart it resumes exactly where it stopped — no duplicates, no gaps.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">f</span><span class="z-entity z-name z-tag">ilebeat.registry.path</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/filebeat/registry</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Internal memory queue — tune for throughput vs. latency.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> These events are lost on crash; durability comes from the registry,</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> not from this queue.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.mem</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> e</span><span class="z-entity z-name z-tag">vents</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 4096</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> f</span><span class="z-entity z-name z-tag">lush.min_events</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 512</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> f</span><span class="z-entity z-name z-tag">lush.timeout</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 5</span><span class="z-string">s</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name z-tag">o</span><span class="z-entity z-name z-tag">utput.logstash</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> h</span><span class="z-entity z-name z-tag">osts</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">logstash:5044</span><span class="z-string">&quot;</span><span>]</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry connecting to Logstash — Filebeat will keep trying indefinitely.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Events accumulate in the queue while Logstash is unreachable.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">imeout</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 30</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> m</span><span class="z-entity z-name z-tag">ax_retries</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 3</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> per batch; 0 = retry forever</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> b</span><span class="z-entity z-name z-tag">ulk_max_size</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 2048</span></span></code></pre> <p><strong>Logstash</strong> (<code>/etc/logstash/logstash.yml</code>):</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Persistent queue — events are written to disk between the input and filter/</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> output stages. Logstash survives restarts without losing in-flight events.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.type</span><span class="z-punctuation z-separator">:</span><span class="z-string"> p</span><span class="z-string">ersisted</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.max_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 1</span><span class="z-string">gb</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.path</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/logstash/queue</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Checkpoint frequency — lower values = more durable, more I/O.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.checkpoint.acks</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1024</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">q</span><span class="z-entity z-name z-tag">ueue.checkpoint.writes</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1024</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Dead-letter queue — events that fail after all retries go here instead</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> of being silently dropped. Inspect with the dead_letter_queue input plugin.</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">d</span><span class="z-entity z-name z-tag">ead_letter_queue.enable</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">d</span><span class="z-entity z-name z-tag">ead_letter_queue.max_bytes</span><span class="z-punctuation z-separator">:</span><span class="z-string"> 1</span><span class="z-string">gb</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">p</span><span class="z-entity z-name z-tag">ath.dead_letter_queue</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">var/lib/logstash/dead_letter_queue</span></span></code></pre> <p><strong>Logstash pipeline</strong> (<code>/etc/logstash/conf.d/nginx-s3.conf</code>):</p> <pre class="giallo z-code"><code data-lang="ruby"><span class="giallo-l"><span>input </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> beats </span><span class="z-punctuation z-section">{</span><span> port </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 5044</span><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>filter </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> No grok, no parsing — pass the raw message through unchanged</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span>output </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> s3 </span><span class="z-punctuation z-section">{</span></span> <span class="giallo-l"><span> bucket </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">my-logs</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> region </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">eu-central-1</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> prefix </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">nginx/%{+YYYY}/%{+MM}/%{+dd}/</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> codec </span><span class="z-punctuation z-separator">=&gt;</span><span> line </span><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> one raw log line per line</span></span> <span class="giallo-l"><span> encoding </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">gzip</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span> size_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 52428800</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> rotate object at 50 MB</span></span> <span class="giallo-l"><span> time_file </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> or after 10 minutes</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Local staging directory — files are assembled here before upload.</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Use a path on a real disk, not tmpfs, for crash safety.</span></span> <span class="giallo-l"><span> temporary_directory </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-string"> &quot;</span><span class="z-string">/var/lib/logstash/s3-tmp</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> Retry failed uploads before giving up and sending to the DLQ.</span></span> <span class="giallo-l"><span> retry_count </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 10</span></span> <span class="giallo-l"><span> retry_delay </span><span class="z-punctuation z-separator">=&gt;</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> seconds between retries</span></span> <span class="giallo-l"><span class="z-punctuation z-section"> }</span></span> <span class="giallo-l"><span class="z-punctuation z-section">}</span></span></code></pre> <p>With <code>queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped.</p> <hr /> <h2 id="running-awffull-against-object-storage-logs">Running AWFFull Against Object Storage Logs<a class="zola-anchor" href="#running-awffull-against-object-storage-logs" aria-label="Anchor link for: running-awffull-against-object-storage-logs">🔗</a></h2> <p><img src="/img/reportflow.drawio.png" alt="AWFFull Report Pipeline — Object Storage → AWFFull → Object Storage" /></p> <div class="my-4 flex flex-col rounded-lg bg-(--admonition-bg)" style="--admonition-bg: var(--admonition-info);" > <div class="flex items-center rounded-t-lg bg-(--admonition-bg) p-1"> <div class="mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert" style="--url: url(icons/info.svg);" > info </div> <span><strong>Info</strong></span> </div> <div class="pl-4"><p>The AWFFull container downloads the log files and the previous <code>awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed.</p> </div> </div> <p>Once logs are stored with date-based prefixes, fetching a specific day is a single command:</p> <p><strong>From S3:</strong></p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Download all log files for 12 March 2026</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/12/</span><span class="z-string"> /tmp/logs/2026-03-12/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Process with AWFFull (pass files in chronological order)</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">ls</span><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-variable z-language">*</span><span class="z-string">.log.gz</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p><strong>From Azure Blob:</strong></p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> &quot;</span><span class="z-string">https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/*</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">ls</span><span class="z-string"> /tmp/logs/2026-03-12/</span><span class="z-variable z-language">*</span><span class="z-string">.log.gz</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p><strong>Downloading a full month</strong> (common for monthly reporting):</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/</span><span class="z-string"> /tmp/logs/2026-03/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.gz</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span> $(</span><span class="z-entity z-name">find</span><span class="z-string"> /tmp/logs/2026-03/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">name</span><span class="z-string"> &quot;</span><span class="z-string">*.log.gz</span><span class="z-string">&quot;</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-entity z-name"> sort</span><span>)</span></span></code></pre> <p>The <code>sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. <code>access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically.</p> <hr /> <h2 id="running-awffull-in-a-container-world">Running AWFFull in a Container World<a class="zola-anchor" href="#running-awffull-in-a-container-world" aria-label="Anchor link for: running-awffull-in-a-container-world">🔗</a></h2> <p>AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns:</p> <hr /> <h3 id="option-1-persistent-volume-classical-approach">Option 1 — Persistent Volume (classical approach)<a class="zola-anchor" href="#option-1-persistent-volume-classical-approach" aria-label="Anchor link for: option-1-persistent-volume-classical-approach">🔗</a></h3> <p>The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup.</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> docker-compose example</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">ervices</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">wffull</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">mage</span><span class="z-punctuation z-separator">:</span><span class="z-string"> r</span><span class="z-string">egistry.gitlab.com/aleks001/awfull:latest</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumes</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">var/log/nginx:/logs:ro</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> log files (read-only)</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">srv/awffull/reports:/reports</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> generated HTML + PNG output</span></span> <span class="giallo-l"><span> -</span><span class="z-string"> /</span><span class="z-string">etc/awffull.conf:/etc/awffull.conf:ro</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ommand</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">-c</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/etc/awffull.conf</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/logs/access.log.gz</span><span class="z-string">&quot;</span><span>]</span></span></code></pre> <p>In Kubernetes, replace the host-path mounts with a <code>PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here.</p> <pre class="giallo z-code"><code data-lang="yaml"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Kubernetes CronJob skeleton</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">a</span><span class="z-entity z-name z-tag">piVersion</span><span class="z-punctuation z-separator">:</span><span class="z-string"> b</span><span class="z-string">atch/v1</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">k</span><span class="z-entity z-name z-tag">ind</span><span class="z-punctuation z-separator">:</span><span class="z-string"> C</span><span class="z-string">ronJob</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">m</span><span class="z-entity z-name z-tag">etadata</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-monthly</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag">s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">chedule</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">0 2 1 * *</span><span class="z-string">&quot;</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> 02:00 on the 1st of each month</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> j</span><span class="z-entity z-name z-tag">obTemplate</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> t</span><span class="z-entity z-name z-tag">emplate</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">pec</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">ontainers</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> i</span><span class="z-entity z-name z-tag">mage</span><span class="z-punctuation z-separator">:</span><span class="z-string"> r</span><span class="z-string">egistry.gitlab.com/aleks001/awfull:latest</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> a</span><span class="z-entity z-name z-tag">rgs</span><span class="z-punctuation z-separator">:</span><span> [</span><span class="z-string">&quot;</span><span class="z-string">-c</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/etc/awffull.conf</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">/logs/access.log.gz</span><span class="z-string">&quot;</span><span>]</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumeMounts</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> l</span><span class="z-entity z-name z-tag">ogs mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">logs</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">eports mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">reports</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onfig mountPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> /</span><span class="z-string">etc/awffull.conf</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> s</span><span class="z-entity z-name z-tag">ubPath</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull.conf</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> v</span><span class="z-entity z-name z-tag">olumes</span><span class="z-punctuation z-separator">:</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> l</span><span class="z-entity z-name z-tag">ogs persistentVolumeClaim</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">laimName</span><span class="z-punctuation z-separator">:</span><span class="z-string"> n</span><span class="z-string">ginx-logs</span><span> }</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">eports persistentVolumeClaim</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">laimName</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-reports</span><span> }</span></span> <span class="giallo-l"><span> -</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-entity z-name z-tag"> c</span><span class="z-entity z-name z-tag">onfig configMap</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-entity z-name z-tag"> n</span><span class="z-entity z-name z-tag">ame</span><span class="z-punctuation z-separator">:</span><span class="z-string"> a</span><span class="z-string">wffull-config</span><span> }</span></span> <span class="giallo-l"><span class="z-entity z-name z-tag"> r</span><span class="z-entity z-name z-tag">estartPolicy</span><span class="z-punctuation z-separator">:</span><span class="z-string"> O</span><span class="z-string">nFailure</span></span></code></pre> <p><strong>Trade-off:</strong> Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod).</p> <hr /> <h3 id="option-2-s3-s3-compatible-object-storage">Option 2 — S3 / S3-compatible Object Storage<a class="zola-anchor" href="#option-2-s3-s3-compatible-object-storage" aria-label="Anchor link for: option-2-s3-s3-compatible-object-storage">🔗</a></h3> <p>This option covers two related questions:</p> <ol> <li>Where do the generated reports go?</li> <li><strong>Can logs that were already shipped to S3 be fed back into AWFFull?</strong></li> </ol> <p>The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline.</p> <p><strong>Shipping logs to S3</strong> is standard practice. Tools like <a rel="noopener external" target="_blank" href="https://fluentbit.io/">Fluent Bit</a>, <a rel="noopener external" target="_blank" href="https://vector.dev/">Vector</a>, or even a plain <code>logrotate</code> + <code>aws s3 cp</code> cron job send compressed log files to a bucket with a path like <code>s3://my-logs/nginx/2026/03/01/access.log.gz</code>.</p> <p><strong>Reading them back</strong> for AWFFull is equally straightforward:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1. Download history file (needed for incremental processing)</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> cp</span><span class="z-string"> s3://my-bucket/awffull/awffull.hist</span><span class="z-string"> /var/lib/awffull/</span><span class="z-keyword z-keyword z-operator"> 2</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-string">/dev/null</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-keyword z-keyword z-operator">|</span><span class="z-support z-support z-function"> true</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2. Download log files for the period to process</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> s3://my-logs/nginx/2026/03/</span><span class="z-string"> /tmp/logs/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.gz</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3. Run AWFFull (reads logs, updates history, writes HTML)</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /tmp/logs/</span><span class="z-variable z-language">*</span><span class="z-string">.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4. Persist the updated history file back to S3</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> cp</span><span class="z-string"> /var/lib/awffull/awffull.hist</span><span class="z-string"> s3://my-bucket/awffull/</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 5. Sync reports to the reports bucket / static website</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-reports-bucket/awffull/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete</span></span></code></pre> <p>The <code>awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (<code>-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well.</p> <p>In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist)</span></span> <span class="giallo-l"><span> ▲</span></span> <span class="giallo-l"><span> └── IAM role / workload identity</span></span></code></pre> <p>Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.).</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> entrypoint wrapper</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /logs/access.log.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Sync output to S3</span></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-bucket/awffull/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-content-type</span><span class="z-string"> &quot;</span><span class="z-string">text/html</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-exclude</span><span class="z-string"> &quot;</span><span class="z-string">*.png</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.html</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-entity z-name">aws</span><span class="z-string"> s3</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> s3://my-bucket/awffull/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-exclude</span><span class="z-string"> &quot;</span><span class="z-string">*.html</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-include</span><span class="z-string"> &quot;</span><span class="z-string">*.png</span><span class="z-string">&quot;</span></span></code></pre> <p>In a container, add <code>awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables.</p> <p><strong>Trade-off:</strong> Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions.</p> <hr /> <h3 id="option-3-azure-blob-storage">Option 3 — Azure Blob Storage<a class="zola-anchor" href="#option-3-azure-blob-storage" aria-label="Anchor link for: option-3-azure-blob-storage">🔗</a></h3> <p>Same pattern as S3, using the Azure CLI (<code>az storage blob</code>) or <code>azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint.</p> <p>Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#!</span><span class="z-comment">/bin/sh</span></span> <span class="giallo-l"><span class="z-variable z-other z-variable">STORAGE</span><span class="z-keyword z-keyword z-operator">=</span><span class="z-string">&quot;</span><span class="z-string">https://&lt;storage-account&gt;.blob.core.windows.net</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1. Download history file</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/awffull/awffull.hist</span><span class="z-string">&quot;</span><span class="z-string"> /var/lib/awffull/</span><span class="z-keyword z-keyword z-operator"> 2</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-string">/dev/null</span><span class="z-keyword z-keyword z-operator"> |</span><span class="z-keyword z-keyword z-operator">|</span><span class="z-support z-support z-function"> true</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2. Download log files for the period</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/nginx-logs/2026/03/*</span><span class="z-string">&quot;</span><span class="z-string"> /tmp/logs/</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3. Run AWFFull</span></span> <span class="giallo-l"><span class="z-entity z-name">awffull</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-string"> /etc/awffull.conf</span><span class="z-string"> /tmp/logs/</span><span class="z-variable z-language">*</span><span class="z-string">.gz</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4. Persist history back to Blob</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> copy</span><span class="z-string"> /var/lib/awffull/awffull.hist</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/awffull/awffull.hist</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 5. Sync reports to static website container ($web)</span></span> <span class="giallo-l"><span class="z-entity z-name">azcopy</span><span class="z-string"> sync</span><span class="z-string"> /reports/</span><span class="z-string"> &quot;</span><span class="z-string">$</span><span class="z-string z-punctuation z-section">{</span><span class="z-variable z-other z-variable">STORAGE</span><span class="z-string z-punctuation z-section">}</span><span class="z-string">/</span><span class="z-constant z-character">\$</span><span class="z-string">web/awffull/</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-recursive</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-delete-destination=true</span></span></code></pre> <p>Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> With workload identity (AKS)</span></span> <span class="giallo-l"><span class="z-entity z-name">az</span><span class="z-string"> login</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-identity</span></span> <span class="giallo-l"><span class="z-entity z-name">az</span><span class="z-string"> storage</span><span class="z-string"> blob</span><span class="z-string"> sync</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-source</span><span class="z-string"> /reports/</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-container</span><span class="z-string"> &#39;</span><span class="z-string">$web</span><span class="z-string">&#39;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-account-name</span><span class="z-keyword z-keyword z-operator"> &lt;</span><span class="z-string">storage-accoun</span><span>t</span><span class="z-keyword z-keyword z-operator">&gt;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-destination</span><span class="z-string"> awffull/</span></span></code></pre> <p>Enable the <strong>Static website</strong> feature on the Storage Account and the reports are immediately reachable at <code>https://&lt;storage-account&gt;.z6.web.core.windows.net/awffull/index.html</code>.</p> <p><strong>Trade-off:</strong> Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials.</p> <hr /> <h3 id="choosing-a-pattern">Choosing a Pattern<a class="zola-anchor" href="#choosing-a-pattern" aria-label="Anchor link for: choosing-a-pattern">🔗</a></h3> <table><thead><tr><th>Scenario</th><th>Recommended approach</th></tr></thead><tbody> <tr><td>Simple on-prem Kubernetes</td><td>Persistent Volume + nginx sidecar</td></tr> <tr><td>AWS / EKS</td><td>S3 + CloudFront static website</td></tr> <tr><td>Azure / AKS</td><td>Azure Blob static website + workload identity</td></tr> <tr><td>Self-hosted, no cloud</td><td>S3 sync</td></tr> <tr><td>CI/CD pipeline artifact</td><td>Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> <p>In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (<code>awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path.</p> <hr /> <p><em>AWFFull is free software licensed under GPL v3 or later.</em> <em>The project is at <a rel="noopener external" target="_blank" href="https://gitlab.com/aleks001/awfful">https://gitlab.com/aleks001/awfful</a>.</em></p>