Format</th>	Magic Bytes</th></tr></thead>
gzip</td>	`1F 8B</code></td></tr>`
bzip2</td>	`42 5A 68</code> (BZh</code>)</td></tr>`
xz</td>	`FD 37 7A 58 5A 00</code></td></tr>`
zstd</td>	28 B5 2F FD</code></td></tr> </tbody></table> pub</span> fn</span> detect_from_bytes</span>(</span>magic</span>:</span> &</span>[</span>u8</span>]</span>)</span> -></span> Compression</span> {</span></span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 2</span> &&</span> magic</span>[</span>..</span>2</span>]</span> ==</span> [</span>0x1f</span>,</span> 0x8b</span>]</span> {</span> Compression</span>::</span>Gz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 3</span> &&</span> magic</span>[</span>..</span>3</span>]</span> ==</span> [</span>0x42</span>,</span> 0x5a</span>,</span> 0x68</span>]</span> {</span> Compression</span>::</span>Bz2</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 6</span> &&</span> magic</span>[</span>..</span>6</span>]</span> ==</span> [</span>0xfd</span>,</span> 0x37</span>,</span> 0x7a</span>,</span> 0x58</span>,</span> 0x5a</span>,</span> 0x00</span>]</span> {</span> Compression</span>::</span>Xz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 4</span> &&</span> magic</span>[</span>..</span>4</span>]</span> ==</span> [</span>0x28</span>,</span> 0xb5</span>,</span> 0x2f</span>,</span> 0xfd</span>]</span> {</span> Compression</span>::</span>Zstd</span> }</span></span> else</span> {</span> Compression</span>::</span>None</span> }</span></span> }</span></span></code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed.</p> 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output.</p> mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone.</p> Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>:</p> ──── profile_timing (41.65M lines) ──────────────────</span></span> I/O (read_line) 3.91 s 32.4%</span></span> Parse (jiff strptime) 5.52 s 45.7% ← bottleneck</span></span> Heap (push/pop) 1.97 s 16.3%</span></span> Write (write_all) 0.69 s 5.7%</span></span> ─────────────────────────────────────────────────</span></span> Σ 12.08 s</span></span></code></pre> Nearly half the time was in the parser.</strong> The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast.</p> This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser.</p> Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed:</p> [DD/Mon/YYYY:HH:MM:SS ±HHMM]</span></span> 0123456789012345678901234 5</span></span></code></pre> A hand-written parser only needs to:</p> Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication:</p> fn</span> days_since_epoch</span>(</span>y</span>:</span> i64</span>,</span> m</span>:</span> i64</span>,</span> d</span>:</span> i64</span>)</span> -></span> i64</span> {</span></span> let</span> y</span> =</span> if</span> m</span> <=</span> 2</span> {</span> y</span> -</span> 1</span> }</span> else</span> {</span> y</span> }</span>;</span></span> let</span> era</span> =</span> if</span> y</span> >=</span> 0</span> {</span> y</span> }</span> else</span> {</span> y</span> -</span> 399</span> }</span> /</span> 400</span>;</span></span> let</span> yoe</span> =</span> y</span> -</span> era</span> </span> 400</span>;</span></span> let</span> doy</span> =</span> (</span>153</span> </span> (</span>m</span> +</span> if</span> m</span> ></span> 2</span> {</span> -</span>3</span> }</span> else</span> {</span> 9</span> }</span>)</span> +</span> 2</span>)</span> /</span> 5</span> +</span> d</span> -</span> 1</span>;</span></span> let</span> doe</span> =</span> yoe</span> </span> 365</span> +</span> yoe</span> /</span> 4</span> -</span> yoe</span> /</span> 100</span> +</span> doy</span>;</span></span> era</span> </span> 146_097</span> +</span> doe</span> -</span> 719_468</span></span> }</span></span></code></pre> Result:</strong> parse time dropped from 5.52 s to 1.50 s — 3.7× faster</strong>.</p> Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read:</p> //</span> before: fresh allocation per line</span></span> fn</span> read_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>)</span> -></span> Result</span><</span>Option</span><</span>String</span>></span>></span> {</span></span> let</span> mut</span> line</span> =</span> String</span>::</span>new</span>(</span>)</span>;</span> //</span> ← new every time</span></span> reader</span>.</span>read_line</span>(</span>&</span>mut</span> line</span>)</span>?</span>;</span></span> ...</span></span> }</span></span></code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer:</p> //</span> after: capacity is retained</span></span> while</span> let</span> Some</span>(</span>Entry</span> {</span> mut</span> line</span>,</span> mut</span> reader</span>,</span> file_idx</span>,</span> ..</span> }</span>)</span> =</span> heap</span>.</span>pop</span>(</span>)</span> {</span></span> out</span>.</span>write_all</span>(</span>line</span>.</span>as_bytes</span>(</span>)</span>)</span>?</span>;</span></span> line</span>.</span>clear</span>(</span>)</span>;</span> //</span> ← keeps the heap allocation</span></span> if</span> fill_line</span>(</span>&</span>mut</span> </span>reader</span>,</span> &</span>mut</span> line</span>)</span>?</span> {</span></span> let</span> ts</span> =</span> parse_clf_timestamp</span>(</span>&</span>line</span>)</span>.</span>unwrap_or_else</span>(</span>sentinel</span>)</span>;</span></span> heap</span>.</span>push</span>(</span>Entry</span> {</span> ts</span>,</span> file_idx</span>,</span> line</span>,</span> reader</span> }</span>)</span>;</span></span> }</span></span> }</span></span></code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path.</p> Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust:</p> #</span> Cargo.toml</span></span> mimalloc</span> =</span> {</span> version</span> =</span> "</span>0.1</span>"</span>,</span> default-features</span> =</span> false</span> }</span></span></code></pre>//</span> main.rs</span></span> use</span> mimalloc</span>::</span>MiMalloc</span>;</span></span> #</span>[</span>global_allocator</span>]</span></span> static</span> GLOBAL</span>:</span> MiMalloc</span> =</span> MiMalloc</span>;</span></span></code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well.</p> Result:</strong> 5.55 s → 5.39 s — a further ~3% improvement.</p> Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else.</p> Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together.</p> 4 MiB read buffers</strong> — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time.</p> SIMD newline search via memchr</code></strong> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>:</p> fn</span> fill_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>,</span> buf</span>:</span> &</span>mut</span> String</span>)</span> -></span> Result</span><</span>bool</span>></span> {</span></span> loop</span> {</span></span> let</span> available</span> =</span> reader</span>.</span>fill_buf</span>(</span>)</span>?</span>;</span></span> if</span> available</span>.</span>is_empty</span>(</span>)</span> {</span> return</span> Ok</span>(</span>false</span>)</span>;</span> }</span></span> match</span> memchr</span>(</span>b</span>'</span>\</span>n</span>'</span>,</span> available</span>)</span> {</span></span> Some</span>(</span>pos</span>)</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>&</span>available</span>[</span>..</span>pos</span> +</span> 1</span>]</span>)</span>?</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>pos</span> +</span> 1</span>)</span>;</span></span> return</span> Ok</span>(</span>true</span>)</span>;</span></span> }</span></span> None</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>available</span>)</span>?</span>)</span>;</span></span> let</span> len</span> =</span> available</span>.</span>len</span>(</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>len</span>)</span>;</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Result:</strong> 5.39 s → 4.43 s — another 18% improvement</strong>, bringing the total to 2.26× faster</strong> than the C original.</p> A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel:</p> Thread 1: read + decompress + parse → channel ─┐</span></span> Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout</span></span> ... ─┘</span></span></code></pre> For compressed files</strong> (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel.</p> For plain-text files</strong> on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower</em> than the sequential version with large buffers.</p> Takeaway:</strong> threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first.</p> Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines:</p> Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</strong></td> 4.43 s</strong></td> 3.61 s</strong></td> 0.81 s</strong></td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</span></span></code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload:</p> </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</strong></td> 1.6 MiB</strong></td> 30.7 MiB</strong></td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory</strong> — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained.</p> System time</strong> — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically.</p> CPU</strong> — Both tools are single-threaded and max out one core at 99%.</p> Takeaways🔗</a></h2> Five lessons from this project:</p> 1. Measure before optimizing.</strong> Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined.</p> 2. Libraries are great until they aren’t.</strong> jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×.</p> 3. Sometimes the allocator is the last few percent.</strong> After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying.</p> 4. Buffer size matters more than you think.</strong> Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better.</p> 5. Parallelism requires the right bottleneck.</strong> Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower</em>. Threading is a tool, not a universal solution.</p> 6. Ownership is not a burden.</strong> Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free.</p> The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this.</p> AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser 2026-03-12T00:00:00+00:00 AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser🔗</a></h1> Background🔗</a></h2> AWFFull 4.0.0</a> is a web server log analyser. It reads Apache/nginx/Squid access logs and produces HTML reports with graphs showing traffic, top URLs, referrers, user agents, country statistics, and more. It was originally forked from The Webalizer</a> by Bradford L. Barrett (1997–2001), extended as AWFFull by Steve McInerney from 2004, and actively developed until 2008, when it was last released as version 3.10.2.</p> It still works well in 2026 — the core log-parsing logic is solid and the output is clean. But the codebase had accumulated 18 years of code: EOL libraries, dead links, obsolete command-line flags, a broken GeoIP implementation, and a very old config.guess</code>. Time for a proper cleanup.</p> This post covers what changed in AWFFull 4.0.0 and why.</p> Is a Static Web Log Analyser Still Useful?🔗</a></h2> Good question. The short answer is: yes — and arguably more so than before.</p> Modern web analytics platforms — Google Analytics, Matomo, Plausible, Grafana with Loki, ELK stacks — are excellent for live dashboards and drill-down exploration. But they share a common weakness: they are slow when you need to process large volumes of historical log data and render the results as a complete set of graphs and tables</strong>.</p> Feed a year of Apache logs (say, 500 million lines across 50 GB of gzipped files) into an ELK pipeline and you’ll be waiting a long time for the data to be ingested, indexed, and ready to query. Then wait again each time you open a dashboard. Grafana rendering a year’s worth of nginx traffic across 30 panels? Expect seconds of load time per page, per query.</p> AWFFull’s approach is different: it processes the raw log files directly, keeps all state in a compact binary history file, and writes static HTML + PNG output</strong> that loads instantly in any browser — even on a machine from 2010, over a slow connection, or offline entirely. There is no query layer, no index, no database, no running server. The report is just files.</p> For the typical use case — a sysadmin or developer who wants a monthly overview of who accessed their server, from where, with what agents, hitting which URLs — this is exactly the right trade-off:</p> Fast to generate</strong>: processing a month of logs takes seconds, not minutes</li> Instant to view</strong>: static HTML, no round-trips, no query engine</li> Self-contained</strong>: the report directory can be archived, moved, or served from any static file server</li> Privacy-friendly</strong>: no third-party JavaScript, no tracking pixels, no data leaving your infrastructure — the raw IP addresses stay in your own logs</li> Works on air-gapped systems</strong>: nothing requires internet access at runtime</li> </ul> It also complements rather than replaces live monitoring. You would typically use Prometheus/Grafana (or similar) to watch your infrastructure in real time, and AWFFull to produce the monthly historical report that goes into a ticket or a management summary.</p> Shipping Access Logs to Object Storage and Running in Containers🔗</a></h3> The operational details — how to ship access logs to S3 or Azure Blob with Fluent Bit, Vector, or Filebeat/Logstash, how to handle concurrent writes and log loss at high traffic, and how to run AWFFull as a stateless container against object storage — are covered in the companion post:</p> AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers</a></strong></p> The Big Ticket Items🔗</a></h2> PCRE → PCRE2🔗</a></h3> AWFFull used libpcre (PCRE version 1) for log-line regex parsing. PCRE 1 has been officially end-of-life since 2012 and is being dropped by distributions. We migrated to PCRE2</strong>, the current maintained successor with a cleaner API and better Unicode support. The change touched parser.c</code>, common.h</code>, and configure.ac</code>.</p> GeoIP: From Legacy .dat</code> to libmaxminddb🔗</a></h3> The old code used libGeoIP</strong> with a proprietary binary .dat</code> format that MaxMind deprecated in 2019. Country lookups were broken for anyone who hadn’t kept a seven-year-old database around.</p> AWFFull 4.0.0 replaces this with libmaxminddb</strong>, the open standard MMDB format used by both MaxMind (GeoLite2) and DB-IP. The migration is transparent — IPv4 and IPv6 are both handled by a single MMDB_lookup_string()</code> call, with no separate code paths needed.</p> Free databases updated monthly:</p> Country</strong>: DB-IP Country Lite</a></li> ASN</strong>: DB-IP ASN Lite</a></li> </ul> ASN Statistics: New Table and Pie Chart🔗</a></h3> Once you have a MMDB-capable lookup layer, adding ASN (Autonomous System Number) statistics is almost free. A second database (GeoASNDatabase</code>) maps IPs to their network operator — think “Cloudflare”, “Amazon AWS”, “Deutsche Telekom”. The report now shows a Top ASNs</strong> table and pie chart, answering the question “which networks send the most traffic to my site?”</em></p> GeoASN yes</span></span> GeoASNDatabase /usr/share/GeoIP/dbip-asn-lite.mmdb</span></span></code></pre> Performance note: both GeoIP and GeoASN lookups happen in the output phase</em>, once per unique IP address — not once per log line. On a typical site the overhead is negligible.</p> Hyperscan: SIMD Pattern Matching🔗</a></h3> AWFFull supports extensive pattern lists for ignoring, grouping, and hiding hosts, URLs, referrers, and agents. The original code used a Boyer-Moore-Horspool</strong> linear scan: each log record was matched against every pattern in turn.</p> When Intel Hyperscan</strong> (libhyperscan</code>) is installed, AWFFull now compiles all patterns from a given list into a single Hyperscan database and matches them in one SIMD pass per record. For large pattern lists — say, 50+ bot/spider agent patterns — this is significantly faster. Hyperscan is auto-detected at configure time and falls back to the existing BMH path if unavailable.</p> sudo</span> apt</span> install</span> libhyperscan-dev</span></span> ./configure</span> &&</span> make</span></span></code></pre>mimalloc: Better Heap Performance🔗</a></h3> For large log files with millions of unique hosts, URLs, and referrers, heap allocation overhead becomes measurable. Adding mimalloc</strong> (Microsoft’s high-performance allocator) as an optional drop-in was straightforward via the explicit mi_calloc</code> API. Auto-detected at configure time.</p> XXH3 Hash Function🔗</a></h3> The internal hash table used djb2, a good general-purpose hash from 1991. AWFFull 4.0.0 replaces it with XXH3</strong> (via the bundled xxhash.h</code>), which offers better distribution and significantly higher throughput on modern CPUs. The table load-factor was also tuned to reduce collisions.</p> User-Visible Improvements🔗</a></h2> Date Stamp on Graphs🔗</a></h3> Every generated PNG graph now shows the current date</strong> (YYYY-MM-DD</code>) in the top-right corner, aligned with the title baseline. Small but useful when you’re looking at archived reports six months later.</p> Next / Previous Month Navigation🔗</a></h3> Monthly report pages now show navigation links at the top:</p> ← February 2026 \| [Index] \| April 2026 →</span></span></code></pre> Links are derived from the history file — only months that actually exist in the database are linked.</p> Updated Browser/Bot Detection🔗</a></h3> The sample.conf</code> browser/agent detection list dated from 2005 and contained entries like “Netscape 4.5”, “Internet Explorer 3.0 (Win95)”, and “MSNBot” — none of which have been seen in the wild for a decade or more. The list has been completely rewritten:</p> Removed:</strong> Netscape 1–4.x, Internet Explorer 3–7, Firebird, Galeon, Camino, msnbot (all entries)</p> Added:</strong> Chrome/Chromium, Edge (Chromium), Brave, Samsung Browser, Firefox, Safari, Opera (OPR/), modern search bots (Bingbot, DuckDuckBot, YandexBot, Baidu, Applebot, AhrefsBot, SemrushBot), AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Amazonbot), and CLI tools (curl, Wget, python-requests).</p> Build System and Tooling🔗</a></h2> CLI Cleanup: 14 Flags Removed🔗</a></h3> AWFFull inherited many single-letter flags from the original Webalizer. Flags like -G</code> (suppress hourly graph), -L</code> (suppress legend), -M</code> (mangle agents), -x</code> (HTML extension), and -I</code> (index alias) had exact equivalents in the config file. At version 4.0.0 — a major version bump — these were removed from the CLI, along with three fully deprecated no-op flags (-d</code>, -q</code>, -Q</code>).</p> The --help</code> output now documents the new verbosity levels (-v</code> through -vvvvv</code>) and includes a note pointing config-file-only settings to awffull.conf(5)</code>.</p> Man Pages: DocBook XML → Pandoc Markdown🔗</a></h3> The existing man pages were written in DocBook XML 4.5 and built with xsltproc</code> — a tool that requires xsltproc</code>, docbook-xsl</code>, and several XML schema packages, none of which are installed by default on most systems in 2026.</p> The man pages are now maintained as pandoc Markdown</strong> (doc/awffull.1.md</code>, doc/awffull.conf.5.md</code>) and built with:</p> pandoc</span> -</span>s</span> -</span>t</span> man</span> awffull.1.md</span> -</span>o</span> awffull.1</span></span></code></pre> configure.ac</code> auto-detects pandoc. Pre-built man pages remain in the repository as a fallback for systems without pandoc. The DocBook XML sources are retained in doc/</code> for historical reference.</p> Translations: Quality Over Quantity🔗</a></h3> AWFFull shipped with 33 language files, most of which were approximately 29% complete and had not been updated since 2008. Rather than shipping incomplete translations that silently fall back to English mid-report, we reduced to 6 fully complete languages</strong> (German, Finnish, Italian, Swedish, Brazilian Portuguese, Indonesian) and brought all of them to 100% (524/524 strings)</strong>.</p> The .pot</code> template was regenerated with xgettext</code> to include all new strings added in 4.0.0 (GeoASN messages, MaxMind DB errors, pattern matching warnings).</p> Repository Modernisation🔗</a></h2> The project moved from a collection of legacy files to a cleaner structure:</p> Before</th> After</th></tr></thead> README</code> + README.FIRST</code> + README.webalizer</code></td> README.md</code> (single entry point)</td></tr> ChangeLog</code> (GNU format, frozen)</td> CHANGELOG.md</code> (Keep a Changelog)</td></tr> COPYING</code></td> LICENSE</code></td></tr> TODO</code> (plain text, 2008)</td> TODO.md</code> (Markdown with [x]</code> checkboxes)</td></tr> country-codes.txt</code> (2005, informational)</td> removed</td></tr> DNS.README</code> (dead link)</td> removed</td></tr> contrib/</code> (Webalizer migration scripts)</td> removed</td></tr> config.guess</code> from 2008</td> updated to automake 1.16 (2022)</td></tr> </tbody></table> tip </div> tip</strong></span> </div> The full list of technical changes is available in CHANGES.md</a>.</p> </div> </div> CI/CD Pipeline🔗</a></h2> A full GitLab CI pipeline was added covering:</p> Build verification</strong> on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 8/9/10</li> Source and binary tarballs</strong> on every push to main</code></li> .deb</code> package</strong> (Ubuntu 24.04 amd64) on release tags</li> .rpm</code> package</strong> (AlmaLinux 9 amd64) on release tags</li> Container image</strong> pushed to GitLab Container Registry (Ubuntu 24.04 base, all optional libs included)</li> GitLab Release</strong> creation with all artifacts uploaded to the Generic Package Registry</li> </ul> Releases are triggered by pushing a semver tag:</p> git</span> tag</span> v4.0.0</span></span> git</span> push</span> -</span>-tags</span></span></code></pre> The container image is available as:</p> docker</span> pull</span> registry.gitlab.com/aleks001/awffull:v4.0.0</span></span> docker</span> pull</span> registry.gitlab.com/aleks001/awffull:latest</span></span></code></pre> The other artifacts are available on the Release Page</a></p> What’s Next🔗</a></h2> The TODO.md</code> tracks longer-term ideas: SVG graphs (replacing the aging libgd dependency), weekly summaries, a templating system for the HTML output, and better browser/OS reporting. Contributions welcome.</p> The project is at https://gitlab.com/aleks001/awfull</a>.</p> AWFFull is free software licensed under GPL v3 or later.</em> Based on The Webalizer by Bradford L. Barrett (1997–2001).</em> Extended by Steve McInerney (2004–2008) and Aleksandar Lazic (2026–present).</em></p> AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers 2026-03-12T00:00:00+00:00 This is a companion post to AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs.</p> Shipping Access Logs to Object Storage🔗</a></h2> Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches.</p> </p> danger </div> Critical requirement for AWFFull compatibility</strong></span> </div> The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON.</p> </div> </div> Fluent Bit → S3🔗</a></h3> Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins.</p> #</span> /etc/fluent-bit/fluent-bit.conf</span></span> </span> [</span>INPUT</span>]</span></span> Name tail</span></span> Path /var/log/nginx/access.log</span></span> Tag nginx.access</span></span> Read_from_Head false</span></span> Refresh_Interval 5</span></span> </span> [</span>OUTPUT</span>]</span></span> Name s3</span></span> Match nginx.access</span></span> bucket my-logs</span></span> region eu-central-1</span></span> #</span> Date-partitioned path — one directory per day.</span></span> #</span> $UUID is a Fluent Bit built-in variable; it ensures unique filenames</span></span> #</span> when multiple instances run concurrently (e.g. multiple Pods).</span></span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> compression gzip</span></span> total_file_size 50M </span>#</span> flush when file reaches 50 MB</span></span> upload_timeout 10m </span>#</span> or after 10 minutes, whichever comes first</span></span> use_put_object On</span></span></code></pre> For Azure Blob</strong> replace the output section:</p> [</span>OUTPUT</span>]</span></span> Name azure_blob</span></span> Match nginx.access</span></span> account_name mystorageaccount</span></span> shared_key ${AZURE_STORAGE_KEY}</span></span> container_name nginx-logs</span></span> #</span> The azure_blob plugin does not support $UUID in the path parameter.</span></span> #</span> Uniqueness is guaranteed because Fluent Bit appends an internal</span></span> #</span> timestamp + sequence counter to each blob name automatically.</span></span> path nginx/%Y/%m/%d</span></span> auto_create_container On</span></span> tls On</span></span> compress gzip</span></span></code></pre> Vector → S3 / Azure Blob🔗</a></h3> Vector</a> offers more flexibility for transformations and supports both sinks natively:</p> #</span> vector.yaml</span></span> </span> s</span>ources</span>:</span></span> n</span>ginx_access</span>:</span></span> t</span>ype</span>:</span> f</span>ile</span></span> i</span>nclude</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> r</span>ead_from</span>:</span> e</span>nd</span> #</span> only new lines; use 'beginning' for backfill</span></span> </span> s</span>inks</span>:</span></span> s</span>3_logs</span>:</span></span> t</span>ype</span>:</span> a</span>ws_s3</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> b</span>ucket</span>:</span> m</span>y-logs</span></span> r</span>egion</span>:</span> e</span>u-central-1</span></span> #</span> Vector automatically appends a UUID to key_prefix — no explicit $UUID needed.</span></span> #</span> Multiple instances writing concurrently will never collide.</span></span> k</span>ey_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> c</span>ompression</span>:</span> g</span>zip</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span> #</span> raw lines — no JSON wrapping</span></span> b</span>atch</span>:</span></span> m</span>ax_bytes</span>:</span> 52428800</span> #</span> 50 MB</span></span> t</span>imeout_secs</span>:</span> 600</span> #</span> or 10 minutes</span></span></code></pre> For Azure Blob, replace the sink:</p> a</span>zure_blob_logs</span>:</span></span> t</span>ype</span>:</span> a</span>zure_blob</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> c</span>onnection_string</span>:</span> "</span>${AZURE_STORAGE_CONNECTION_STRING}</span>"</span></span> c</span>ontainer_name</span>:</span> n</span>ginx-logs</span></span> #</span> Same as S3: Vector appends a UUID automatically — uniqueness is built-in.</span></span> b</span>lob_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span></span> c</span>ompression</span>:</span> g</span>zip</span></span></code></pre> Filebeat → Logstash → S3🔗</a></h3> Filebeat</a> is the standard shipper in the Elastic Stack. It has an nginx module</a> that parses, enriches and structures the access log — but do not use the nginx module for AWFFull</strong>. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format.</p> Instead, use the generic filestream</code> input with no parsing:</p> #</span> /etc/filebeat/filebeat.yml</span></span> </span> f</span>ilebeat.inputs</span>:</span></span> -</span> t</span>ype</span>:</span> f</span>ilestream</span></span> i</span>d</span>:</span> n</span>ginx-access</span></span> p</span>aths</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> #</span> No parsers, no module — raw lines only</span></span> p</span>arsers</span>:</span> [</span>]</span></span> </span> o</span>utput.logstash</span>:</span></span> h</span>osts</span>:</span> [</span>"</span>logstash:5044</span>"</span>]</span></span></code></pre> Filebeat has no native S3 output. Use Logstash</strong> as the intermediary with the logstash-output-s3</code> plugin:</p> #</span> /etc/logstash/conf.d/nginx-s3.conf</span></span> </span> input </span>{</span></span> beats </span>{</span> port </span>=></span> 5044</span> }</span></span> }</span></span> </span> filter </span>{</span></span> #</span> No grok, no parsing — pass the raw message through unchanged</span></span> }</span></span> </span> output </span>{</span></span> s3 </span>{</span></span> bucket </span>=></span> "</span>my-logs</span>"</span></span> region </span>=></span> "</span>eu-central-1</span>"</span></span> prefix </span>=></span> "</span>nginx/%{+YYYY}/%{+MM}/%{+dd}/</span>"</span></span> codec </span>=></span> line </span>#</span> raw lines, one per file</span></span> encoding </span>=></span> "</span>gzip</span>"</span></span> size_file </span>=></span> 52428800</span> #</span> 50 MB per file</span></span> time_file </span>=></span> 10</span> #</span> or rotate after 10 minutes</span></span> #</span> Credentials from IAM role / environment</span></span> }</span></span> }</span></span></code></pre> The codec => line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default json_lines</code> codec would break compatibility.</p> Note:</strong> Filebeat also supports shipping to Azure via Logstash with the logstash-output-azure_blob_storage</a> community plugin, using the same pattern.</p> Concurrent Write Safety🔗</a></h3> When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: can this corrupt the data?</strong></p> Short answer: no — but data loss is possible if you don’t use unique keys.</strong></p> S3 and Azure Blob both guarantee atomic PUT semantics</strong> at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible.</p> The real risk is silent overwrite</strong>: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost.</p> Here is how the tools described above handle this:</p> Tool</th> Mechanism</th> Multiple instances</th></tr></thead> Fluent Bit → S3</td> $UUID</code> in s3_key_format</code></td> Safe — each instance generates a unique key</td></tr> Vector → S3</td> UUID appended automatically to key_prefix</code></td> Safe — built-in</td></tr> Vector → Azure Blob</td> UUID appended automatically to blob_prefix</code></td> Safe — built-in</td></tr> Logstash → S3</td> hostname + timestamp</code> in the generated filename</td> Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> Fluent Bit → Azure Blob (append blob)</td> Append blob type with atomic server-side appends</td> Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> Fluent Bit → Azure Blob (block blob)</td> Path parameter sets the blob name</td> Unsafe if multiple instances use the same path</code> without a unique suffix</td></tr> </tbody></table> Practical rule:</strong> always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key.</p> Log Loss at High-Traffic Sites🔗</a></h3> Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are:</p> 1. Memory buffer overflow</strong> Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either drops new lines silently</strong> or applies backpressure</strong> upstream — which can slow down the web server itself.</p> 2. Network outage or S3/Azure throttling</strong> Object storage APIs return 503 Slow Down</code> (S3) or 429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload.</p> 3. Shipper crash without offset tracking</strong> If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage).</p> 4. Log rotation race</strong> logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost.</p> Each tool has a different answer to these problems:</p> Tool</th> Persistence</th> At-least-once guarantee</th></tr></thead> Fluent Bit</td> Filesystem buffer (storage.type filesystem</code>)</td> Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> Vector</td> Disk buffer (type: disk</code> in buffer config)</td> Yes — survives restarts; default is in-memory only</td></tr> Logstash</td> Persistent queue (queue.type: persisted</code>)</td> Yes — survives restarts; default is in-memory only</td></tr> Filebeat</td> Registry file tracks read offset per file</td> Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated and deleted</strong> before Filebeat reads to the end, the unread bytes are lost.</p> Recommendations for high-traffic sites:</strong></p> Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> Set a generous retry policy and back-off interval for S3/Azure upload errors</li> Configure log rotation with a delaycompress</code> / rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool.</p> Fluent Bit — Filesystem Buffer + Retry🔗</a></h4> #</span> /etc/fluent-bit/fluent-bit.conf</span></span> </span> [</span>SERVICE</span>]</span></span> Flush 5</span></span> Daemon Off</span></span> Log_Level info</span></span> </span> #</span> Filesystem buffering — chunks are written to disk before upload.</span></span> #</span> Data survives Fluent Bit restarts and network outages up to the limit.</span></span> storage.path /var/lib/fluent-bit/buf/</span></span> storage.sync normal </span>#</span> fsync after each chunk write</span></span> storage.checksum off</span></span> storage.max_chunks_up 128 </span>#</span> max chunks in memory at once</span></span> </span> [</span>INPUT</span>]</span></span> Name tail</span></span> Path /var/log/nginx/access.log</span></span> Tag nginx.access</span></span> Read_from_Head false</span></span> Refresh_Interval 5</span></span> #</span> Offset database — Fluent Bit remembers the last read position.</span></span> #</span> Without this, every restart re-reads the file from the beginning.</span></span> DB /var/lib/fluent-bit/nginx-access.db</span></span> #</span> Use filesystem storage for this input's chunks (not memory)</span></span> storage.type filesystem</span></span> </span> [</span>OUTPUT</span>]</span></span> Name s3</span></span> Match nginx.access</span></span> bucket my-logs</span></span> region eu-central-1</span></span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> compression gzip</span></span> total_file_size 50M</span></span> upload_timeout 10m</span></span> use_put_object On</span></span> </span> #</span> Retry: up to 5 attempts with exponential back-off (max ~160 s between tries).</span></span> #</span> Set False for unlimited retries (use with caution — can block the pipeline).</span></span> Retry_Limit 5</span></span> </span> #</span> Cap the on-disk buffer for this output at 1 GB.</span></span> #</span> When full, oldest chunks are dropped to make room.</span></span> storage.total_limit_size 1G</span></span></code></pre> The DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if logrotate</code> renames access.log</code> to access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file.</p> Vector — Disk Buffer + Retry🔗</a></h4> #</span> /etc/vector/vector.yaml</span></span> </span> s</span>ources</span>:</span></span> n</span>ginx_access</span>:</span></span> t</span>ype</span>:</span> f</span>ile</span></span> i</span>nclude</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> r</span>ead_from</span>:</span> e</span>nd</span></span> #</span> Vector also writes a checkpoint file (similar to Filebeat's registry)</span></span> #</span> to /var/lib/vector/ by default — survives restarts automatically.</span></span> </span> s</span>inks</span>:</span></span> s</span>3_logs</span>:</span></span> t</span>ype</span>:</span> a</span>ws_s3</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> b</span>ucket</span>:</span> m</span>y-logs</span></span> r</span>egion</span>:</span> e</span>u-central-1</span></span> k</span>ey_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> c</span>ompression</span>:</span> g</span>zip</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span></span> b</span>atch</span>:</span></span> m</span>ax_bytes</span>:</span> 52428800</span> #</span> 50 MB per object</span></span> t</span>imeout_secs</span>:</span> 600</span> #</span> or 10 minutes, whichever comes first</span></span> </span> #</span> Persistent disk buffer — events are written to disk before upload.</span></span> #</span> Survives Vector restarts; default is type: memory (lost on crash).</span></span> b</span>uffer</span>:</span></span> t</span>ype</span>:</span> d</span>isk</span></span> m</span>ax_size</span>:</span> 1073741824</span> #</span> 1 GB</span></span> w</span>hen_full</span>:</span> b</span>lock</span> #</span> apply backpressure instead of dropping events</span></span> </span> #</span> Retry policy for S3 upload failures (503, 429, network errors).</span></span> r</span>equest</span>:</span></span> r</span>etry_attempts</span>:</span> 10</span> #</span> max upload attempts per batch</span></span> r</span>etry_initial_backoff_secs</span>:</span> 1</span> #</span> first retry after 1 s</span></span> r</span>etry_max_duration_secs</span>:</span> 300</span> #</span> cap back-off at 5 minutes</span></span> t</span>imeout_secs</span>:</span> 60</span> #</span> per-request timeout</span></span> r</span>ate_limit_num</span>:</span> 100</span> #</span> max concurrent requests</span></span></code></pre> when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog.</p> Filebeat + Logstash — Registry + Persistent Queue🔗</a></h4> Filebeat</strong> (/etc/filebeat/filebeat.yml</code>):</p> f</span>ilebeat.inputs</span>:</span></span> -</span> t</span>ype</span>:</span> f</span>ilestream</span></span> i</span>d</span>:</span> n</span>ginx-access</span></span> p</span>aths</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> p</span>arsers</span>:</span> [</span>]</span> #</span> no parsing — raw lines only</span></span> </span> #</span> Filebeat writes a registry to /var/lib/filebeat/registry/ that records</span></span> #</span> the inode, byte offset, and identifier for every tracked file.</span></span> #</span> On restart it resumes exactly where it stopped — no duplicates, no gaps.</span></span> f</span>ilebeat.registry.path</span>:</span> /</span>var/lib/filebeat/registry</span></span> </span> #</span> Internal memory queue — tune for throughput vs. latency.</span></span> #</span> These events are lost on crash; durability comes from the registry,</span></span> #</span> not from this queue.</span></span> q</span>ueue.mem</span>:</span></span> e</span>vents</span>:</span> 4096</span></span> f</span>lush.min_events</span>:</span> 512</span></span> f</span>lush.timeout</span>:</span> 5</span>s</span></span> </span> o</span>utput.logstash</span>:</span></span> h</span>osts</span>:</span> [</span>"</span>logstash:5044</span>"</span>]</span></span> #</span> Retry connecting to Logstash — Filebeat will keep trying indefinitely.</span></span> #</span> Events accumulate in the queue while Logstash is unreachable.</span></span> t</span>imeout</span>:</span> 30</span></span> m</span>ax_retries</span>:</span> 3</span> #</span> per batch; 0 = retry forever</span></span> b</span>ulk_max_size</span>:</span> 2048</span></span></code></pre> Logstash</strong> (/etc/logstash/logstash.yml</code>):</p> #</span> Persistent queue — events are written to disk between the input and filter/</span></span> #</span> output stages. Logstash survives restarts without losing in-flight events.</span></span> q</span>ueue.type</span>:</span> p</span>ersisted</span></span> q</span>ueue.max_bytes</span>:</span> 1</span>gb</span></span> q</span>ueue.path</span>:</span> /</span>var/lib/logstash/queue</span></span> </span> #</span> Checkpoint frequency — lower values = more durable, more I/O.</span></span> q</span>ueue.checkpoint.acks</span>:</span> 1024</span></span> q</span>ueue.checkpoint.writes</span>:</span> 1024</span></span> </span> #</span> Dead-letter queue — events that fail after all retries go here instead</span></span> #</span> of being silently dropped. Inspect with the dead_letter_queue input plugin.</span></span> d</span>ead_letter_queue.enable</span>:</span> true</span></span> d</span>ead_letter_queue.max_bytes</span>:</span> 1</span>gb</span></span> p</span>ath.dead_letter_queue</span>:</span> /</span>var/lib/logstash/dead_letter_queue</span></span></code></pre> Logstash pipeline</strong> (/etc/logstash/conf.d/nginx-s3.conf</code>):</p> input </span>{</span></span> beats </span>{</span> port </span>=></span> 5044</span> }</span></span> }</span></span> </span> filter </span>{</span></span> #</span> No grok, no parsing — pass the raw message through unchanged</span></span> }</span></span> </span> output </span>{</span></span> s3 </span>{</span></span> bucket </span>=></span> "</span>my-logs</span>"</span></span> region </span>=></span> "</span>eu-central-1</span>"</span></span> prefix </span>=></span> "</span>nginx/%{+YYYY}/%{+MM}/%{+dd}/</span>"</span></span> codec </span>=></span> line </span>#</span> one raw log line per line</span></span> encoding </span>=></span> "</span>gzip</span>"</span></span> size_file </span>=></span> 52428800</span> #</span> rotate object at 50 MB</span></span> time_file </span>=></span> 10</span> #</span> or after 10 minutes</span></span> #</span> Local staging directory — files are assembled here before upload.</span></span> #</span> Use a path on a real disk, not tmpfs, for crash safety.</span></span> temporary_directory </span>=></span> "</span>/var/lib/logstash/s3-tmp</span>"</span></span> #</span> Retry failed uploads before giving up and sending to the DLQ.</span></span> retry_count </span>=></span> 10</span></span> retry_delay </span>=></span> 1</span> #</span> seconds between retries</span></span> }</span></span> }</span></span></code></pre> With queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped.</p> Running AWFFull Against Object Storage Logs🔗</a></h2> </p> info </div> Info</strong></span> </div> The AWFFull container downloads the log files and the previous awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed.</p> </div> </div> Once logs are stored with date-based prefixes, fetching a specific day is a single command:</p> From S3:</strong></p> #</span> Download all log files for 12 March 2026</span></span> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/12/</span> /tmp/logs/2026-03-12/</span></span> </span> #</span> Process with AWFFull (pass files in chronological order)</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>ls</span> /tmp/logs/2026-03-12/</span></span>.log.gz</span> \|</span> sort</span>)</span></span></code></pre> From Azure Blob:</strong></p> azcopy</span> copy</span> \</span></span> "</span>https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/</span>"</span> \</span></span> /tmp/logs/2026-03-12/</span> \</span></span> -</span>-recursive</span></span> </span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>ls</span> /tmp/logs/2026-03-12/</span></span>.log.gz</span> \|</span> sort</span>)</span></span></code></pre> Downloading a full month</strong> (common for monthly reporting):</p> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/</span> /tmp/logs/2026-03/</span> -</span>-include</span> "</span>.gz</span>"</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>find</span> /tmp/logs/2026-03/</span> -</span>name</span> "</span>.log.gz</span>"</span> \|</span> sort</span>)</span></span></code></pre> The sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically.</p> Running AWFFull in a Container World🔗</a></h2> AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns:</p> Option 1 — Persistent Volume (classical approach)🔗</a></h3> The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup.</p> #</span> docker-compose example</span></span> s</span>ervices</span>:</span></span> a</span>wffull</span>:</span></span> i</span>mage</span>:</span> r</span>egistry.gitlab.com/aleks001/awfull:latest</span></span> v</span>olumes</span>:</span></span> -</span> /</span>var/log/nginx:/logs:ro</span> #</span> log files (read-only)</span></span> -</span> /</span>srv/awffull/reports:/reports</span> #</span> generated HTML + PNG output</span></span> -</span> /</span>etc/awffull.conf:/etc/awffull.conf:ro</span></span> c</span>ommand</span>:</span> [</span>"</span>-c</span>"</span>,</span> "</span>/etc/awffull.conf</span>"</span>,</span> "</span>/logs/access.log.gz</span>"</span>]</span></span></code></pre> In Kubernetes, replace the host-path mounts with a PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here.</p> #</span> Kubernetes CronJob skeleton</span></span> a</span>piVersion</span>:</span> b</span>atch/v1</span></span> k</span>ind</span>:</span> C</span>ronJob</span></span> m</span>etadata</span>:</span></span> n</span>ame</span>:</span> a</span>wffull-monthly</span></span> s</span>pec</span>:</span></span> s</span>chedule</span>:</span> "</span>0 2 1 * </span>"</span> #</span> 02:00 on the 1st of each month</span></span> j</span>obTemplate</span>:</span></span> s</span>pec</span>:</span></span> t</span>emplate</span>:</span></span> s</span>pec</span>:</span></span> c</span>ontainers</span>:</span></span> -</span> n</span>ame</span>:</span> a</span>wffull</span></span> i</span>mage</span>:</span> r</span>egistry.gitlab.com/aleks001/awfull:latest</span></span> a</span>rgs</span>:</span> [</span>"</span>-c</span>"</span>,</span> "</span>/etc/awffull.conf</span>"</span>,</span> "</span>/logs/access.log.gz</span>"</span>]</span></span> v</span>olumeMounts</span>:</span></span> -</span> n</span>ame</span>:</span> l</span>ogs mountPath</span>:</span> /</span>logs</span></span> -</span> n</span>ame</span>:</span> r</span>eports mountPath</span>:</span> /</span>reports</span></span> -</span> n</span>ame</span>:</span> c</span>onfig mountPath</span>:</span> /</span>etc/awffull.conf</span></span> s</span>ubPath</span>:</span> a</span>wffull.conf</span></span> v</span>olumes</span>:</span></span> -</span> n</span>ame</span>:</span> l</span>ogs persistentVolumeClaim</span>:</span> {</span> c</span>laimName</span>:</span> n</span>ginx-logs</span> }</span></span> -</span> n</span>ame</span>:</span> r</span>eports persistentVolumeClaim</span>:</span> {</span> c</span>laimName</span>:</span> a</span>wffull-reports</span> }</span></span> -</span> n</span>ame</span>:</span> c</span>onfig configMap</span>:</span> {</span> n</span>ame</span>:</span> a</span>wffull-config</span> }</span></span> r</span>estartPolicy</span>:</span> O</span>nFailure</span></span></code></pre> Trade-off:</strong> Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod).</p> Option 2 — S3 / S3-compatible Object Storage🔗</a></h3> This option covers two related questions:</p> Where do the generated reports go?</li> Can logs that were already shipped to S3 be fed back into AWFFull?</strong></li> </ol> The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline.</p> Shipping logs to S3</strong> is standard practice. Tools like Fluent Bit</a>, Vector</a>, or even a plain logrotate</code> + aws s3 cp</code> cron job send compressed log files to a bucket with a path like s3://my-logs/nginx/2026/03/01/access.log.gz</code>.</p> Reading them back</strong> for AWFFull is equally straightforward:</p> #!</span>/bin/sh</span></span> #</span> 1. Download history file (needed for incremental processing)</span></span> aws</span> s3</span> cp</span> s3://my-bucket/awffull/awffull.hist</span> /var/lib/awffull/</span> 2</span>></span>/dev/null</span> \|</span>\|</span> true</span></span> </span> #</span> 2. Download log files for the period to process</span></span> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/</span> /tmp/logs/</span> -</span>-include</span> "</span>.gz</span>"</span></span> </span> #</span> 3. Run AWFFull (reads logs, updates history, writes HTML)</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /tmp/logs/</span></span>.gz</span></span> </span> #</span> 4. Persist the updated history file back to S3</span></span> aws</span> s3</span> cp</span> /var/lib/awffull/awffull.hist</span> s3://my-bucket/awffull/</span></span> </span> #</span> 5. Sync reports to the reports bucket / static website</span></span> aws</span> s3</span> sync</span> /reports/</span> s3://my-reports-bucket/awffull/</span> -</span>-delete</span></span></code></pre> The awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well.</p> In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all:</p> S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist)</span></span> ▲</span></span> └── IAM role / workload identity</span></span></code></pre> Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.).</p> #!</span>/bin/sh</span></span> #</span> entrypoint wrapper</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /logs/access.log.gz</span></span> </span> #</span> Sync output to S3</span></span> aws</span> s3</span> sync</span> /reports/</span> s3://my-bucket/awffull/</span> \</span></span> -</span>-delete</span> \</span></span> -</span>-content-type</span> "</span>text/html</span>"</span> \</span></span> -</span>-exclude</span> "</span>.png</span>"</span> \</span></span> -</span>-include</span> "</span>.html</span>"</span></span> </span> aws</span> s3</span> sync</span> /reports/</span> s3://my-bucket/awffull/</span> \</span></span> -</span>-exclude</span> "</span>.html</span>"</span> \</span></span> -</span>-include</span> "</span>.png</span>"</span></span></code></pre> In a container, add awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables.</p> Trade-off:</strong> Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions.</p> Option 3 — Azure Blob Storage🔗</a></h3> Same pattern as S3, using the Azure CLI (az storage blob</code>) or azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint.</p> Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull:</p> #!</span>/bin/sh</span></span> STORAGE</span>=</span>"</span>https://<storage-account>.blob.core.windows.net</span>"</span></span> </span> #</span> 1. Download history file</span></span> azcopy</span> copy</span> "</span>$</span>{</span>STORAGE</span>}</span>/awffull/awffull.hist</span>"</span> /var/lib/awffull/</span> 2</span>></span>/dev/null</span> \|</span>\|</span> true</span></span> </span> #</span> 2. Download log files for the period</span></span> azcopy</span> copy</span> "</span>$</span>{</span>STORAGE</span>}</span>/nginx-logs/2026/03/</span>"</span> /tmp/logs/</span> -</span>-recursive</span></span> </span> #</span> 3. Run AWFFull</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /tmp/logs/</span>*</span>.gz</span></span> </span> #</span> 4. Persist history back to Blob</span></span> azcopy</span> copy</span> /var/lib/awffull/awffull.hist</span> "</span>$</span>{</span>STORAGE</span>}</span>/awffull/awffull.hist</span>"</span></span> </span> #</span> 5. Sync reports to static website container ($web)</span></span> azcopy</span> sync</span> /reports/</span> "</span>$</span>{</span>STORAGE</span>}</span>/</span>\$</span>web/awffull/</span>"</span> \</span></span> -</span>-recursive</span> \</span></span> -</span>-delete-destination=true</span></span></code></pre> Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image:</p> #</span> With workload identity (AKS)</span></span> az</span> login</span> -</span>-identity</span></span> az</span> storage</span> blob</span> sync</span> \</span></span> -</span>-source</span> /reports/</span> \</span></span> -</span>-container</span> '</span>$web</span>'</span> \</span></span> -</span>-account-name</span> <</span>storage-accoun</span>t</span>></span> \</span></span> -</span>-destination</span> awffull/</span></span></code></pre> Enable the Static website</strong> feature on the Storage Account and the reports are immediately reachable at https://<storage-account>.z6.web.core.windows.net/awffull/index.html</code>.</p> Trade-off:</strong> Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials.</p> Choosing a Pattern🔗</a></h3> Scenario</th> Recommended approach</th></tr></thead> Simple on-prem Kubernetes</td> Persistent Volume + nginx sidecar</td></tr> AWS / EKS</td> S3 + CloudFront static website</td></tr> Azure / AKS</td> Azure Blob static website + workload identity</td></tr> Self-hosted, no cloud</td> S3 sync</td></tr> CI/CD pipeline artifact</td> Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path.</p> AWFFull is free software licensed under GPL v3 or later.</em> The project is at https://gitlab.com/aleks001/awfful</a>.</em></p>

mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster

2026-03-15T00:00:00+00:00

mergelog</a> is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. The original version 4.5 was written in C around 2000–2001, weighs in at ~450 lines, and does exactly what it promises — reliably and fast.

This post documents the journey from mergelog-4.5 to mergelog-rs</a>: what changed, what was surprising, and how six targeted optimizations made the Rust binary 2.26× faster than the C original.

Full respect and honour to Bertrand Demiddelaer, who created mergelog back in 2000. Writing a tool this fast, this correct, and this compact in C — with no external dependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his shoulders.

The Original: mergelog 4.5 🔗</a></h2>
The C program reads any number of log files in Common Log Format (CLF) / NCSA Combined Format:
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"</code></pre> Example line:93.184.216.34 - - [15/Mar/2026:12:00:00 +0100] "GET /index.html HTTP/1.1" 200 1234 "-" "curl/7.88"</code></pre> The algorithm in 4.5 is straightforward: find the earliest timestamp across all files, then iterate second-by-second from there to the end, writing all matching lines at each step. That is O(timespan × K) — slow when logs span multiple years or many files are involved. Gzip support exists, but as a separate binary (zmergelog</code>) compiled with -DUSE_ZLIB</code>. The Rust Rewrite: What Changed🔗</a></h2> 1. Algorithm: K-Way Merge with a Binary Heap🔗</a></h3> Instead of time-based iteration, mergelog-rs uses a k-way merge with a min-heap (std::collections::BinaryHeap</code> with reversed ordering): Seed the heap with the first line from each file</li> Main loop: pop the smallest timestamp, write the line, push the next line from the same file</li> Complexity: O(N log K) — independent of the time span of the logs</li> </ul> while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre>2. Magic-Byte Detection Instead of File Extension🔗</a></h3> The original detects gzip by binary type (zmergelog</code> vs. mergelog</code>). mergelog-rs reads the first 6 bytes of each file and detects the format automatically: Format</th> Magic Bytes</th></tr></thead> gzip</td> 1F 8B</code></td></tr> bzip2</td> 42 5A 68</code> (BZh</code>)</td></tr> xz</td> FD 37 7A 58 5A 00</code></td></tr> zstd</td> 28 B5 2F FD</code></td></tr> </tbody></table> pub fn detect_from_bytes(magic: &[u8]) -> Compression { if magic.len() >= 2 && magic[..2] == [0x1f, 0x8b] { Compression::Gz } else if magic.len() >= 3 && magic[..3] == [0x42, 0x5a, 0x68] { Compression::Bz2 } else if magic.len() >= 6 && magic[..6] == [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00] { Compression::Xz } else if magic.len() >= 4 && magic[..4] == [0x28, 0xb5, 0x2f, 0xfd] { Compression::Zstd } else { Compression::None } }</code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed. 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output. mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone. Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>: ──── profile_timing (41.65M lines) ────────────────── I/O (read_line) 3.91 s 32.4% Parse (jiff strptime) 5.52 s 45.7% ← bottleneck Heap (push/pop) 1.97 s 16.3% Write (write_all) 0.69 s 5.7% ───────────────────────────────────────────────── Σ 12.08 s</code></pre> Nearly half the time was in the parser. The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast. This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser. Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed: [DD/Mon/YYYY:HH:MM:SS ±HHMM] 0123456789012345678901234 5</code></pre> A hand-written parser only needs to: Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication: fn days_since_epoch(y: i64, m: i64, d: i64) -> i64 { let y = if m <= 2 { y - 1 } else { y }; let era = if y >= 0 { y } else { y - 399 } / 400; let yoe = y - era * 400; let doy = (153 * (m + if m > 2 { -3 } else { 9 }) + 2) / 5 + d - 1; let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; era * 146_097 + doe - 719_468 }</code></pre> Result: parse time dropped from 5.52 s to 1.50 s — 3.7× faster. Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read: // before: fresh allocation per line fn read_line(reader: &mut dyn BufRead) -> Result<Option<String>> { let mut line = String::new(); // ← new every time reader.read_line(&mut line)?; ... }</code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer: // after: capacity is retained while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); // ← keeps the heap allocation if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path. Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust: # Cargo.toml mimalloc = { version = "0.1", default-features = false }</code></pre>// main.rs use mimalloc::MiMalloc; #[global_allocator] static GLOBAL: MiMalloc = MiMalloc;</code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well. Result: 5.55 s → 5.39 s — a further ~3% improvement. Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else. Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together. 4 MiB read buffers — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time. SIMD newline search via memchr</code> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>: fn fill_line(reader: &mut dyn BufRead, buf: &mut String) -> Result<bool> { loop { let available = reader.fill_buf()?; if available.is_empty() { return Ok(false); } match memchr(b'\n', available) { Some(pos) => { buf.push_str(std::str::from_utf8(&available[..pos + 1])?); reader.consume(pos + 1); return Ok(true); } None => { buf.push_str(std::str::from_utf8(available)?); let len = available.len(); reader.consume(len); } } } }</code></pre> Result: 5.39 s → 4.43 s — another 18% improvement, bringing the total to 2.26× faster than the C original. A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel: Thread 1: read + decompress + parse → channel ─┐ Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout ... ─┘</code></pre> For compressed files (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel. For plain-text files on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower than the sequential version with large buffers. Takeaway: threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first. Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines: Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</td> 4.43 s</td> 3.61 s</td> 0.81 s</td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload: </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</td> 1.6 MiB</td> 30.7 MiB</td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained. System time — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically. CPU — Both tools are single-threaded and max out one core at 99%. Takeaways🔗</a></h2> Five lessons from this project: 1. Measure before optimizing. Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined. 2. Libraries are great until they aren’t. jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×. 3. Sometimes the allocator is the last few percent. After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying. 4. Buffer size matters more than you think. Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better. 5. Parallelism requires the right bottleneck. Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower. Threading is a tool, not a universal solution. 6. Ownership is not a burden. Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free. The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this. AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser 2026-03-12T00:00:00+00:00 AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser🔗</a></h1> Background🔗</a></h2> AWFFull 4.0.0</a> is a web server log analyser. It reads Apache/nginx/Squid access logs and produces HTML reports with graphs showing traffic, top URLs, referrers, user agents, country statistics, and more. It was originally forked from The Webalizer</a> by Bradford L. Barrett (1997–2001), extended as AWFFull by Steve McInerney from 2004, and actively developed until 2008, when it was last released as version 3.10.2. It still works well in 2026 — the core log-parsing logic is solid and the output is clean. But the codebase had accumulated 18 years of code: EOL libraries, dead links, obsolete command-line flags, a broken GeoIP implementation, and a very old config.guess</code>. Time for a proper cleanup. This post covers what changed in AWFFull 4.0.0 and why. Is a Static Web Log Analyser Still Useful?🔗</a></h2> Good question. The short answer is: yes — and arguably more so than before. Modern web analytics platforms — Google Analytics, Matomo, Plausible, Grafana with Loki, ELK stacks — are excellent for live dashboards and drill-down exploration. But they share a common weakness: they are slow when you need to process large volumes of historical log data and render the results as a complete set of graphs and tables. Feed a year of Apache logs (say, 500 million lines across 50 GB of gzipped files) into an ELK pipeline and you’ll be waiting a long time for the data to be ingested, indexed, and ready to query. Then wait again each time you open a dashboard. Grafana rendering a year’s worth of nginx traffic across 30 panels? Expect seconds of load time per page, per query. AWFFull’s approach is different: it processes the raw log files directly, keeps all state in a compact binary history file, and writes static HTML + PNG output that loads instantly in any browser — even on a machine from 2010, over a slow connection, or offline entirely. There is no query layer, no index, no database, no running server. The report is just files. For the typical use case — a sysadmin or developer who wants a monthly overview of who accessed their server, from where, with what agents, hitting which URLs — this is exactly the right trade-off: Fast to generate: processing a month of logs takes seconds, not minutes</li> Instant to view: static HTML, no round-trips, no query engine</li> Self-contained: the report directory can be archived, moved, or served from any static file server</li> Privacy-friendly: no third-party JavaScript, no tracking pixels, no data leaving your infrastructure — the raw IP addresses stay in your own logs</li> Works on air-gapped systems: nothing requires internet access at runtime</li> </ul> It also complements rather than replaces live monitoring. You would typically use Prometheus/Grafana (or similar) to watch your infrastructure in real time, and AWFFull to produce the monthly historical report that goes into a ticket or a management summary. Shipping Access Logs to Object Storage and Running in Containers🔗</a></h3> The operational details — how to ship access logs to S3 or Azure Blob with Fluent Bit, Vector, or Filebeat/Logstash, how to handle concurrent writes and log loss at high traffic, and how to run AWFFull as a stateless container against object storage — are covered in the companion post: AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers</a> The Big Ticket Items🔗</a></h2> PCRE → PCRE2🔗</a></h3> AWFFull used libpcre (PCRE version 1) for log-line regex parsing. PCRE 1 has been officially end-of-life since 2012 and is being dropped by distributions. We migrated to PCRE2, the current maintained successor with a cleaner API and better Unicode support. The change touched parser.c</code>, common.h</code>, and configure.ac</code>. GeoIP: From Legacy .dat</code> to libmaxminddb🔗</a></h3> The old code used libGeoIP with a proprietary binary .dat</code> format that MaxMind deprecated in 2019. Country lookups were broken for anyone who hadn’t kept a seven-year-old database around. AWFFull 4.0.0 replaces this with libmaxminddb, the open standard MMDB format used by both MaxMind (GeoLite2) and DB-IP. The migration is transparent — IPv4 and IPv6 are both handled by a single MMDB_lookup_string()</code> call, with no separate code paths needed. Free databases updated monthly: Country: DB-IP Country Lite</a></li> ASN: DB-IP ASN Lite</a></li> </ul> ASN Statistics: New Table and Pie Chart🔗</a></h3> Once you have a MMDB-capable lookup layer, adding ASN (Autonomous System Number) statistics is almost free. A second database (GeoASNDatabase</code>) maps IPs to their network operator — think “Cloudflare”, “Amazon AWS”, “Deutsche Telekom”. The report now shows a Top ASNs table and pie chart, answering the question “which networks send the most traffic to my site?” GeoASN yes GeoASNDatabase /usr/share/GeoIP/dbip-asn-lite.mmdb</code></pre> Performance note: both GeoIP and GeoASN lookups happen in the output phase, once per unique IP address — not once per log line. On a typical site the overhead is negligible. Hyperscan: SIMD Pattern Matching🔗</a></h3> AWFFull supports extensive pattern lists for ignoring, grouping, and hiding hosts, URLs, referrers, and agents. The original code used a Boyer-Moore-Horspool linear scan: each log record was matched against every pattern in turn. When Intel Hyperscan (libhyperscan</code>) is installed, AWFFull now compiles all patterns from a given list into a single Hyperscan database and matches them in one SIMD pass per record. For large pattern lists — say, 50+ bot/spider agent patterns — this is significantly faster. Hyperscan is auto-detected at configure time and falls back to the existing BMH path if unavailable. sudo apt install libhyperscan-dev ./configure && make</code></pre>mimalloc: Better Heap Performance🔗</a></h3> For large log files with millions of unique hosts, URLs, and referrers, heap allocation overhead becomes measurable. Adding mimalloc (Microsoft’s high-performance allocator) as an optional drop-in was straightforward via the explicit mi_calloc</code> API. Auto-detected at configure time. XXH3 Hash Function🔗</a></h3> The internal hash table used djb2, a good general-purpose hash from 1991. AWFFull 4.0.0 replaces it with XXH3 (via the bundled xxhash.h</code>), which offers better distribution and significantly higher throughput on modern CPUs. The table load-factor was also tuned to reduce collisions. User-Visible Improvements🔗</a></h2> Date Stamp on Graphs🔗</a></h3> Every generated PNG graph now shows the current date (YYYY-MM-DD</code>) in the top-right corner, aligned with the title baseline. Small but useful when you’re looking at archived reports six months later. Next / Previous Month Navigation🔗</a></h3> Monthly report pages now show navigation links at the top: ← February 2026 | [Index] | April 2026 →</code></pre> Links are derived from the history file — only months that actually exist in the database are linked. Updated Browser/Bot Detection🔗</a></h3> The sample.conf</code> browser/agent detection list dated from 2005 and contained entries like “Netscape 4.5”, “Internet Explorer 3.0 (Win95)”, and “MSNBot” — none of which have been seen in the wild for a decade or more. The list has been completely rewritten: Removed: Netscape 1–4.x, Internet Explorer 3–7, Firebird, Galeon, Camino, msnbot (all entries) Added: Chrome/Chromium, Edge (Chromium), Brave, Samsung Browser, Firefox, Safari, Opera (OPR/), modern search bots (Bingbot, DuckDuckBot, YandexBot, Baidu, Applebot, AhrefsBot, SemrushBot), AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Amazonbot), and CLI tools (curl, Wget, python-requests). Build System and Tooling🔗</a></h2> CLI Cleanup: 14 Flags Removed🔗</a></h3> AWFFull inherited many single-letter flags from the original Webalizer. Flags like -G</code> (suppress hourly graph), -L</code> (suppress legend), -M</code> (mangle agents), -x</code> (HTML extension), and -I</code> (index alias) had exact equivalents in the config file. At version 4.0.0 — a major version bump — these were removed from the CLI, along with three fully deprecated no-op flags (-d</code>, -q</code>, -Q</code>). The --help</code> output now documents the new verbosity levels (-v</code> through -vvvvv</code>) and includes a note pointing config-file-only settings to awffull.conf(5)</code>. Man Pages: DocBook XML → Pandoc Markdown🔗</a></h3> The existing man pages were written in DocBook XML 4.5 and built with xsltproc</code> — a tool that requires xsltproc</code>, docbook-xsl</code>, and several XML schema packages, none of which are installed by default on most systems in 2026. The man pages are now maintained as pandoc Markdown (doc/awffull.1.md</code>, doc/awffull.conf.5.md</code>) and built with: pandoc -s -t man awffull.1.md -o awffull.1</code></pre> configure.ac</code> auto-detects pandoc. Pre-built man pages remain in the repository as a fallback for systems without pandoc. The DocBook XML sources are retained in doc/</code> for historical reference. Translations: Quality Over Quantity🔗</a></h3> AWFFull shipped with 33 language files, most of which were approximately 29% complete and had not been updated since 2008. Rather than shipping incomplete translations that silently fall back to English mid-report, we reduced to 6 fully complete languages (German, Finnish, Italian, Swedish, Brazilian Portuguese, Indonesian) and brought all of them to 100% (524/524 strings). The .pot</code> template was regenerated with xgettext</code> to include all new strings added in 4.0.0 (GeoASN messages, MaxMind DB errors, pattern matching warnings). Repository Modernisation🔗</a></h2> The project moved from a collection of legacy files to a cleaner structure: Before</th> After</th></tr></thead> README</code> + README.FIRST</code> + README.webalizer</code></td> README.md</code> (single entry point)</td></tr> ChangeLog</code> (GNU format, frozen)</td> CHANGELOG.md</code> (Keep a Changelog)</td></tr> COPYING</code></td> LICENSE</code></td></tr> TODO</code> (plain text, 2008)</td> TODO.md</code> (Markdown with [x]</code> checkboxes)</td></tr> country-codes.txt</code> (2005, informational)</td> removed</td></tr> DNS.README</code> (dead link)</td> removed</td></tr> contrib/</code> (Webalizer migration scripts)</td> removed</td></tr> config.guess</code> from 2008</td> updated to automake 1.16 (2022)</td></tr> </tbody></table> tip </div> tip </div> The full list of technical changes is available in CHANGES.md</a>. </div> </div> CI/CD Pipeline🔗</a></h2> A full GitLab CI pipeline was added covering: Build verification on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 8/9/10</li> Source and binary tarballs on every push to main</code></li> .deb</code> package (Ubuntu 24.04 amd64) on release tags</li> .rpm</code> package (AlmaLinux 9 amd64) on release tags</li> Container image pushed to GitLab Container Registry (Ubuntu 24.04 base, all optional libs included)</li> GitLab Release creation with all artifacts uploaded to the Generic Package Registry</li> </ul> Releases are triggered by pushing a semver tag: git tag v4.0.0 git push --tags</code></pre> The container image is available as: docker pull registry.gitlab.com/aleks001/awffull:v4.0.0 docker pull registry.gitlab.com/aleks001/awffull:latest</code></pre> The other artifacts are available on the Release Page</a> What’s Next🔗</a></h2> The TODO.md</code> tracks longer-term ideas: SVG graphs (replacing the aging libgd dependency), weekly summaries, a templating system for the HTML output, and better browser/OS reporting. Contributions welcome. The project is at https://gitlab.com/aleks001/awfull</a>. AWFFull is free software licensed under GPL v3 or later. Based on The Webalizer by Bradford L. Barrett (1997–2001). Extended by Steve McInerney (2004–2008) and Aleksandar Lazic (2026–present). AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers 2026-03-12T00:00:00+00:00 This is a companion post to AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs. Shipping Access Logs to Object Storage🔗</a></h2> Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches. danger </div> Critical requirement for AWFFull compatibility </div> The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON. </div> </div> Fluent Bit → S3🔗</a></h3> Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins. # /etc/fluent-bit/fluent-bit.conf [INPUT] Name tail Path /var/log/nginx/access.log Tag nginx.access Read_from_Head false Refresh_Interval 5 [OUTPUT] Name s3 Match nginx.access bucket my-logs region eu-central-1 # Date-partitioned path — one directory per day. # $UUID is a Fluent Bit built-in variable; it ensures unique filenames # when multiple instances run concurrently (e.g. multiple Pods). s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz compression gzip total_file_size 50M # flush when file reaches 50 MB upload_timeout 10m # or after 10 minutes, whichever comes first use_put_object On</code></pre> For Azure Blob replace the output section: [OUTPUT] Name azure_blob Match nginx.access account_name mystorageaccount shared_key ${AZURE_STORAGE_KEY} container_name nginx-logs # The azure_blob plugin does not support $UUID in the path parameter. # Uniqueness is guaranteed because Fluent Bit appends an internal # timestamp + sequence counter to each blob name automatically. path nginx/%Y/%m/%d auto_create_container On tls On compress gzip</code></pre> Vector → S3 / Azure Blob🔗</a></h3> Vector</a> offers more flexibility for transformations and supports both sinks natively: # vector.yaml sources: nginx_access: type: file include: - /var/log/nginx/access.log read_from: end # only new lines; use 'beginning' for backfill sinks: s3_logs: type: aws_s3 inputs: [nginx_access] bucket: my-logs region: eu-central-1 # Vector automatically appends a UUID to key_prefix — no explicit $UUID needed. # Multiple instances writing concurrently will never collide. key_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" compression: gzip encoding: codec: text # raw lines — no JSON wrapping batch: max_bytes: 52428800 # 50 MB timeout_secs: 600 # or 10 minutes</code></pre> For Azure Blob, replace the sink: azure_blob_logs: type: azure_blob inputs: [nginx_access] connection_string: "${AZURE_STORAGE_CONNECTION_STRING}" container_name: nginx-logs # Same as S3: Vector appends a UUID automatically — uniqueness is built-in. blob_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" encoding: codec: text compression: gzip</code></pre> Filebeat → Logstash → S3🔗</a></h3> Filebeat</a> is the standard shipper in the Elastic Stack. It has an nginx module</a> that parses, enriches and structures the access log — but do not use the nginx module for AWFFull. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format. Instead, use the generic filestream</code> input with no parsing: # /etc/filebeat/filebeat.yml filebeat.inputs: - type: filestream id: nginx-access paths: - /var/log/nginx/access.log # No parsers, no module — raw lines only parsers: [] output.logstash: hosts: ["logstash:5044"]</code></pre> Filebeat has no native S3 output. Use Logstash as the intermediary with the logstash-output-s3</code> plugin: # /etc/logstash/conf.d/nginx-s3.conf input { beats { port => 5044 } } filter { # No grok, no parsing — pass the raw message through unchanged } output { s3 { bucket => "my-logs" region => "eu-central-1" prefix => "nginx/%{+YYYY}/%{+MM}/%{+dd}/" codec => line # raw lines, one per file encoding => "gzip" size_file => 52428800 # 50 MB per file time_file => 10 # or rotate after 10 minutes # Credentials from IAM role / environment } }</code></pre> The codec => line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default json_lines</code> codec would break compatibility. Note: Filebeat also supports shipping to Azure via Logstash with the logstash-output-azure_blob_storage</a> community plugin, using the same pattern. Concurrent Write Safety🔗</a></h3> When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: can this corrupt the data? Short answer: no — but data loss is possible if you don’t use unique keys. S3 and Azure Blob both guarantee atomic PUT semantics at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible. The real risk is silent overwrite: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost. Here is how the tools described above handle this: Tool</th> Mechanism</th> Multiple instances</th></tr></thead> Fluent Bit → S3</td> $UUID</code> in s3_key_format</code></td> Safe — each instance generates a unique key</td></tr> Vector → S3</td> UUID appended automatically to key_prefix</code></td> Safe — built-in</td></tr> Vector → Azure Blob</td> UUID appended automatically to blob_prefix</code></td> Safe — built-in</td></tr> Logstash → S3</td> hostname + timestamp</code> in the generated filename</td> Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> Fluent Bit → Azure Blob (append blob)</td> Append blob type with atomic server-side appends</td> Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> Fluent Bit → Azure Blob (block blob)</td> Path parameter sets the blob name</td> Unsafe if multiple instances use the same path</code> without a unique suffix</td></tr> </tbody></table> Practical rule: always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key. Log Loss at High-Traffic Sites🔗</a></h3> Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are: 1. Memory buffer overflow Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either drops new lines silently or applies backpressure upstream — which can slow down the web server itself. 2. Network outage or S3/Azure throttling Object storage APIs return 503 Slow Down</code> (S3) or 429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload. 3. Shipper crash without offset tracking If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage). 4. Log rotation race logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost. Each tool has a different answer to these problems: Tool</th> Persistence</th> At-least-once guarantee</th></tr></thead> Fluent Bit</td> Filesystem buffer (storage.type filesystem</code>)</td> Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> Vector</td> Disk buffer (type: disk</code> in buffer config)</td> Yes — survives restarts; default is in-memory only</td></tr> Logstash</td> Persistent queue (queue.type: persisted</code>)</td> Yes — survives restarts; default is in-memory only</td></tr> Filebeat</td> Registry file tracks read offset per file</td> Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated and deleted before Filebeat reads to the end, the unread bytes are lost. Recommendations for high-traffic sites: Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> Set a generous retry policy and back-off interval for S3/Azure upload errors</li> Configure log rotation with a delaycompress</code> / rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool. Fluent Bit — Filesystem Buffer + Retry🔗</a></h4> # /etc/fluent-bit/fluent-bit.conf [SERVICE] Flush 5 Daemon Off Log_Level info # Filesystem buffering — chunks are written to disk before upload. # Data survives Fluent Bit restarts and network outages up to the limit. storage.path /var/lib/fluent-bit/buf/ storage.sync normal # fsync after each chunk write storage.checksum off storage.max_chunks_up 128 # max chunks in memory at once [INPUT] Name tail Path /var/log/nginx/access.log Tag nginx.access Read_from_Head false Refresh_Interval 5 # Offset database — Fluent Bit remembers the last read position. # Without this, every restart re-reads the file from the beginning. DB /var/lib/fluent-bit/nginx-access.db # Use filesystem storage for this input's chunks (not memory) storage.type filesystem [OUTPUT] Name s3 Match nginx.access bucket my-logs region eu-central-1 s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz compression gzip total_file_size 50M upload_timeout 10m use_put_object On # Retry: up to 5 attempts with exponential back-off (max ~160 s between tries). # Set False for unlimited retries (use with caution — can block the pipeline). Retry_Limit 5 # Cap the on-disk buffer for this output at 1 GB. # When full, oldest chunks are dropped to make room. storage.total_limit_size 1G</code></pre> The DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if logrotate</code> renames access.log</code> to access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file. Vector — Disk Buffer + Retry🔗</a></h4> # /etc/vector/vector.yaml sources: nginx_access: type: file include: - /var/log/nginx/access.log read_from: end # Vector also writes a checkpoint file (similar to Filebeat's registry) # to /var/lib/vector/ by default — survives restarts automatically. sinks: s3_logs: type: aws_s3 inputs: [nginx_access] bucket: my-logs region: eu-central-1 key_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" compression: gzip encoding: codec: text batch: max_bytes: 52428800 # 50 MB per object timeout_secs: 600 # or 10 minutes, whichever comes first # Persistent disk buffer — events are written to disk before upload. # Survives Vector restarts; default is type: memory (lost on crash). buffer: type: disk max_size: 1073741824 # 1 GB when_full: block # apply backpressure instead of dropping events # Retry policy for S3 upload failures (503, 429, network errors). request: retry_attempts: 10 # max upload attempts per batch retry_initial_backoff_secs: 1 # first retry after 1 s retry_max_duration_secs: 300 # cap back-off at 5 minutes timeout_secs: 60 # per-request timeout rate_limit_num: 100 # max concurrent requests</code></pre> when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog. Filebeat + Logstash — Registry + Persistent Queue🔗</a></h4> Filebeat (/etc/filebeat/filebeat.yml</code>): filebeat.inputs: - type: filestream id: nginx-access paths: - /var/log/nginx/access.log parsers: [] # no parsing — raw lines only # Filebeat writes a registry to /var/lib/filebeat/registry/ that records # the inode, byte offset, and identifier for every tracked file. # On restart it resumes exactly where it stopped — no duplicates, no gaps. filebeat.registry.path: /var/lib/filebeat/registry # Internal memory queue — tune for throughput vs. latency. # These events are lost on crash; durability comes from the registry, # not from this queue. queue.mem: events: 4096 flush.min_events: 512 flush.timeout: 5s output.logstash: hosts: ["logstash:5044"] # Retry connecting to Logstash — Filebeat will keep trying indefinitely. # Events accumulate in the queue while Logstash is unreachable. timeout: 30 max_retries: 3 # per batch; 0 = retry forever bulk_max_size: 2048</code></pre> Logstash (/etc/logstash/logstash.yml</code>): # Persistent queue — events are written to disk between the input and filter/ # output stages. Logstash survives restarts without losing in-flight events. queue.type: persisted queue.max_bytes: 1gb queue.path: /var/lib/logstash/queue # Checkpoint frequency — lower values = more durable, more I/O. queue.checkpoint.acks: 1024 queue.checkpoint.writes: 1024 # Dead-letter queue — events that fail after all retries go here instead # of being silently dropped. Inspect with the dead_letter_queue input plugin. dead_letter_queue.enable: true dead_letter_queue.max_bytes: 1gb path.dead_letter_queue: /var/lib/logstash/dead_letter_queue</code></pre> Logstash pipeline (/etc/logstash/conf.d/nginx-s3.conf</code>): input { beats { port => 5044 } } filter { # No grok, no parsing — pass the raw message through unchanged } output { s3 { bucket => "my-logs" region => "eu-central-1" prefix => "nginx/%{+YYYY}/%{+MM}/%{+dd}/" codec => line # one raw log line per line encoding => "gzip" size_file => 52428800 # rotate object at 50 MB time_file => 10 # or after 10 minutes # Local staging directory — files are assembled here before upload. # Use a path on a real disk, not tmpfs, for crash safety. temporary_directory => "/var/lib/logstash/s3-tmp" # Retry failed uploads before giving up and sending to the DLQ. retry_count => 10 retry_delay => 1 # seconds between retries } }</code></pre> With queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped. Running AWFFull Against Object Storage Logs🔗</a></h2> info </div> Info </div> The AWFFull container downloads the log files and the previous awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed. </div> </div> Once logs are stored with date-based prefixes, fetching a specific day is a single command: From S3: # Download all log files for 12 March 2026 aws s3 sync s3://my-logs/nginx/2026/03/12/ /tmp/logs/2026-03-12/ # Process with AWFFull (pass files in chronological order) awffull -c /etc/awffull.conf $(ls /tmp/logs/2026-03-12/*.log.gz | sort)</code></pre> From Azure Blob: azcopy copy \ "https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/*" \ /tmp/logs/2026-03-12/ \ --recursive awffull -c /etc/awffull.conf $(ls /tmp/logs/2026-03-12/*.log.gz | sort)</code></pre> Downloading a full month (common for monthly reporting): aws s3 sync s3://my-logs/nginx/2026/03/ /tmp/logs/2026-03/ --include "*.gz" awffull -c /etc/awffull.conf $(find /tmp/logs/2026-03/ -name "*.log.gz" | sort)</code></pre> The sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically. Running AWFFull in a Container World🔗</a></h2> AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns: Option 1 — Persistent Volume (classical approach)🔗</a></h3> The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup. # docker-compose example services: awffull: image: registry.gitlab.com/aleks001/awfull:latest volumes: - /var/log/nginx:/logs:ro # log files (read-only) - /srv/awffull/reports:/reports # generated HTML + PNG output - /etc/awffull.conf:/etc/awffull.conf:ro command: ["-c", "/etc/awffull.conf", "/logs/access.log.gz"]</code></pre> In Kubernetes, replace the host-path mounts with a PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here. # Kubernetes CronJob skeleton apiVersion: batch/v1 kind: CronJob metadata: name: awffull-monthly spec: schedule: "0 2 1 * *" # 02:00 on the 1st of each month jobTemplate: spec: template: spec: containers: - name: awffull image: registry.gitlab.com/aleks001/awfull:latest args: ["-c", "/etc/awffull.conf", "/logs/access.log.gz"] volumeMounts: - name: logs mountPath: /logs - name: reports mountPath: /reports - name: config mountPath: /etc/awffull.conf subPath: awffull.conf volumes: - name: logs persistentVolumeClaim: { claimName: nginx-logs } - name: reports persistentVolumeClaim: { claimName: awffull-reports } - name: config configMap: { name: awffull-config } restartPolicy: OnFailure</code></pre> Trade-off: Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod). Option 2 — S3 / S3-compatible Object Storage🔗</a></h3> This option covers two related questions: Where do the generated reports go?</li> Can logs that were already shipped to S3 be fed back into AWFFull?</li> </ol> The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline. Shipping logs to S3 is standard practice. Tools like Fluent Bit</a>, Vector</a>, or even a plain logrotate</code> + aws s3 cp</code> cron job send compressed log files to a bucket with a path like s3://my-logs/nginx/2026/03/01/access.log.gz</code>. Reading them back for AWFFull is equally straightforward: #!/bin/sh # 1. Download history file (needed for incremental processing) aws s3 cp s3://my-bucket/awffull/awffull.hist /var/lib/awffull/ 2>/dev/null || true # 2. Download log files for the period to process aws s3 sync s3://my-logs/nginx/2026/03/ /tmp/logs/ --include "*.gz" # 3. Run AWFFull (reads logs, updates history, writes HTML) awffull -c /etc/awffull.conf /tmp/logs/*.gz # 4. Persist the updated history file back to S3 aws s3 cp /var/lib/awffull/awffull.hist s3://my-bucket/awffull/ # 5. Sync reports to the reports bucket / static website aws s3 sync /reports/ s3://my-reports-bucket/awffull/ --delete</code></pre> The awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well. In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all: S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist) ▲ └── IAM role / workload identity</code></pre> Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.). #!/bin/sh # entrypoint wrapper awffull -c /etc/awffull.conf /logs/access.log.gz # Sync output to S3 aws s3 sync /reports/ s3://my-bucket/awffull/ \ --delete \ --content-type "text/html" \ --exclude "*.png" \ --include "*.html" aws s3 sync /reports/ s3://my-bucket/awffull/ \ --exclude "*.html" \ --include "*.png"</code></pre> In a container, add awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables. Trade-off: Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions. Option 3 — Azure Blob Storage🔗</a></h3> Same pattern as S3, using the Azure CLI (az storage blob</code>) or azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint. Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull: #!/bin/sh STORAGE="https://<storage-account>.blob.core.windows.net" # 1. Download history file azcopy copy "${STORAGE}/awffull/awffull.hist" /var/lib/awffull/ 2>/dev/null || true # 2. Download log files for the period azcopy copy "${STORAGE}/nginx-logs/2026/03/*" /tmp/logs/ --recursive # 3. Run AWFFull awffull -c /etc/awffull.conf /tmp/logs/*.gz # 4. Persist history back to Blob azcopy copy /var/lib/awffull/awffull.hist "${STORAGE}/awffull/awffull.hist" # 5. Sync reports to static website container ($web) azcopy sync /reports/ "${STORAGE}/\$web/awffull/" \ --recursive \ --delete-destination=true</code></pre> Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image: # With workload identity (AKS) az login --identity az storage blob sync \ --source /reports/ \ --container '$web' \ --account-name <storage-account> \ --destination awffull/</code></pre> Enable the Static website feature on the Storage Account and the reports are immediately reachable at https://<storage-account>.z6.web.core.windows.net/awffull/index.html</code>. Trade-off: Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials. Choosing a Pattern🔗</a></h3> Scenario</th> Recommended approach</th></tr></thead> Simple on-prem Kubernetes</td> Persistent Volume + nginx sidecar</td></tr> AWS / EKS</td> S3 + CloudFront static website</td></tr> Azure / AKS</td> Azure Blob static website + workload identity</td></tr> Self-hosted, no cloud</td> S3 sync</td></tr> CI/CD pipeline artifact</td> Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path. AWFFull is free software licensed under GPL v3 or later. The project is at https://gitlab.com/aleks001/awfful</a>.

AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser

AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers

Alek's Blog - nginx

ClickHouse Use Case: Vector, Istio, nginx, Java on OVH MKS

mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster