<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
    <title>Alek&#x27;s Blog - apache</title>
    <subtitle>Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty.</subtitle>
    <link rel="self" type="application/atom+xml" href="https://blog.none.at/tags/apache/atom.xml"/>
    <link rel="alternate" type="text/html" href="https://blog.none.at"/>
    <generator uri="https://www.getzola.org/">Zola</generator>
    <updated>2026-03-15T00:00:00+00:00</updated>
    <id>https://blog.none.at/tags/apache/atom.xml</id>
    <entry xml:lang="en">
        <title>mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster</title>
        <published>2026-03-15T00:00:00+00:00</published>
        <updated>2026-03-15T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              aleks
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://blog.none.at/blog/2026/2026-03-15-mergelog-rs/"/>
        <id>https://blog.none.at/blog/2026/2026-03-15-mergelog-rs/</id>
        
        <content type="html" xml:base="https://blog.none.at/blog/2026/2026-03-15-mergelog-rs/">&lt;p&gt;&lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;mergelog.sourceforge.net&#x2F;&quot;&gt;mergelog&lt;&#x2F;a&gt; is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. The original version 4.5 was written in C around 2000–2001, weighs in at ~450 lines, and does exactly what it promises — reliably and fast.&lt;&#x2F;p&gt;
&lt;p&gt;This post documents the journey from mergelog-4.5 to &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;git001&#x2F;mergelog-rs&quot;&gt;mergelog-rs&lt;&#x2F;a&gt;: what changed, what was surprising, and how six targeted optimizations made the Rust binary &lt;strong&gt;2.26× faster&lt;&#x2F;strong&gt; than the C original.&lt;&#x2F;p&gt;
&lt;p&gt;Full respect and honour to &lt;strong&gt;Bertrand Demiddelaer&lt;&#x2F;strong&gt;, who created mergelog back in 2000. Writing a tool this fast, this correct, and this compact in C — with no external dependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his shoulders.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;the-original-mergelog-4-5&quot;&gt;The Original: mergelog 4.5&lt;a class=&quot;zola-anchor&quot; href=&quot;#the-original-mergelog-4-5&quot; aria-label=&quot;Anchor link for: the-original-mergelog-4-5&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;The C program reads any number of log files in &lt;em&gt;Common Log Format&lt;&#x2F;em&gt; (CLF) &#x2F; &lt;em&gt;NCSA Combined Format&lt;&#x2F;em&gt;:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;%h %l %u %t &amp;quot;%r&amp;quot; %&amp;gt;s %b &amp;quot;%{Referer}i&amp;quot; &amp;quot;%{User-agent}i&amp;quot;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Example line:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;93.184.216.34 - - [15&#x2F;Mar&#x2F;2026:12:00:00 +0100] &amp;quot;GET &#x2F;index.html HTTP&#x2F;1.1&amp;quot; 200 1234 &amp;quot;-&amp;quot; &amp;quot;curl&#x2F;7.88&amp;quot;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;The algorithm in 4.5 is straightforward: find the earliest timestamp across all files, then iterate second-by-second from there to the end, writing all matching lines at each step. That is &lt;strong&gt;O(timespan × K)&lt;&#x2F;strong&gt; — slow when logs span multiple years or many files are involved.&lt;&#x2F;p&gt;
&lt;p&gt;Gzip support exists, but as a separate binary (&lt;code&gt;zmergelog&lt;&#x2F;code&gt;) compiled with &lt;code&gt;-DUSE_ZLIB&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;the-rust-rewrite-what-changed&quot;&gt;The Rust Rewrite: What Changed&lt;a class=&quot;zola-anchor&quot; href=&quot;#the-rust-rewrite-what-changed&quot; aria-label=&quot;Anchor link for: the-rust-rewrite-what-changed&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;h3 id=&quot;1-algorithm-k-way-merge-with-a-binary-heap&quot;&gt;1. Algorithm: K-Way Merge with a Binary Heap&lt;a class=&quot;zola-anchor&quot; href=&quot;#1-algorithm-k-way-merge-with-a-binary-heap&quot; aria-label=&quot;Anchor link for: 1-algorithm-k-way-merge-with-a-binary-heap&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;Instead of time-based iteration, mergelog-rs uses a &lt;strong&gt;k-way merge with a min-heap&lt;&#x2F;strong&gt; (&lt;code&gt;std::collections::BinaryHeap&lt;&#x2F;code&gt; with reversed ordering):&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;Seed the heap with the first line from each file&lt;&#x2F;li&gt;
&lt;li&gt;Main loop: pop the smallest timestamp, write the line, push the next line from the same file&lt;&#x2F;li&gt;
&lt;li&gt;Complexity: &lt;strong&gt;O(N log K)&lt;&#x2F;strong&gt; — independent of the time span of the logs&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;while&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt; let&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Some&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Entry&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt; mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt; mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; reader&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; file_idx&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ..&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; heap&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;pop&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    out&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;write_all&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;line&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;as_bytes&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    line&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;clear&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    if&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; fill_line&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;reader&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;        let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; ts&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; parse_clf_timestamp&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;line&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;unwrap_or_else&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;sentinel&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;        heap&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;push&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Entry&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; ts&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; file_idx&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; reader&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;&lt;h3 id=&quot;2-magic-byte-detection-instead-of-file-extension&quot;&gt;2. Magic-Byte Detection Instead of File Extension&lt;a class=&quot;zola-anchor&quot; href=&quot;#2-magic-byte-detection-instead-of-file-extension&quot; aria-label=&quot;Anchor link for: 2-magic-byte-detection-instead-of-file-extension&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The original detects gzip by binary type (&lt;code&gt;zmergelog&lt;&#x2F;code&gt; vs. &lt;code&gt;mergelog&lt;&#x2F;code&gt;). mergelog-rs reads the first 6 bytes of each file and detects the format automatically:&lt;&#x2F;p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Format&lt;&#x2F;th&gt;&lt;th&gt;Magic Bytes&lt;&#x2F;th&gt;&lt;&#x2F;tr&gt;&lt;&#x2F;thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;gzip&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;1F 8B&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;bzip2&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;42 5A 68&lt;&#x2F;code&gt; (&lt;code&gt;BZh&lt;&#x2F;code&gt;)&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;xz&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;FD 37 7A 58 5A 00&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;zstd&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;28 B5 2F FD&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;&#x2F;tbody&gt;&lt;&#x2F;table&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;pub&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; fn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; detect_from_bytes&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;magic&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;u8&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&amp;gt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 2&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;..&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;2&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ==&lt;&#x2F;span&gt;&lt;span&gt; [&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;0x1f&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x8b&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Gz&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    else&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 3&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;..&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;3&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ==&lt;&#x2F;span&gt;&lt;span&gt; [&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;0x42&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x5a&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x68&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Bz2&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    else&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 6&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;..&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;6&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ==&lt;&#x2F;span&gt;&lt;span&gt; [&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;0xfd&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x37&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x7a&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x58&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x5a&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x00&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Xz&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    else&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 4&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; magic&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;..&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;4&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ==&lt;&#x2F;span&gt;&lt;span&gt; [&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;0x28&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0xb5&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0x2f&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0xfd&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Zstd&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    else&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Compression&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;None&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;One binary, all formats, no file extension required. The same peek-based detection also works for stdin — &lt;code&gt;BufReader::fill_buf()&lt;&#x2F;code&gt; inspects the first bytes without consuming them, so no seek is needed.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;3-timezone-aware-timestamp-comparison&quot;&gt;3. Timezone-Aware Timestamp Comparison&lt;a class=&quot;zola-anchor&quot; href=&quot;#3-timezone-aware-timestamp-comparison&quot; aria-label=&quot;Anchor link for: 3-timezone-aware-timestamp-comparison&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The C original ignores the timezone offset in timestamps — it converts date components to local time via &lt;code&gt;mktime()&lt;&#x2F;code&gt;. For logs from servers in different timezones this can produce incorrectly ordered output.&lt;&#x2F;p&gt;
&lt;p&gt;mergelog-rs converts every timestamp to UTC before comparison. A &lt;code&gt;+0900&lt;&#x2F;code&gt; timestamp and a &lt;code&gt;-0500&lt;&#x2F;code&gt; timestamp are ordered correctly, regardless of the system’s local timezone.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;profiling-where-does-the-time-go&quot;&gt;Profiling: Where Does the Time Go?&lt;a class=&quot;zola-anchor&quot; href=&quot;#profiling-where-does-the-time-go&quot; aria-label=&quot;Anchor link for: profiling-where-does-the-time-go&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;Since &lt;code&gt;perf&lt;&#x2F;code&gt; was locked down on the system (&lt;code&gt;perf_event_paranoid = 4&lt;&#x2F;code&gt;), a dedicated timing binary was built that measures each phase manually with &lt;code&gt;std::time::Instant&lt;&#x2F;code&gt;:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;──── profile_timing (41.65M lines) ──────────────────&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  I&#x2F;O   (read_line)       3.91 s   32.4%&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  Parse (jiff strptime)   5.52 s   45.7%  ← bottleneck&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  Heap  (push&#x2F;pop)        1.97 s   16.3%&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  Write (write_all)       0.69 s    5.7%&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  ─────────────────────────────────────────────────&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;  Σ                      12.08 s&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;strong&gt;Nearly half the time was in the parser.&lt;&#x2F;strong&gt; The reason: &lt;code&gt;jiff::Zoned::strptime&lt;&#x2F;code&gt; is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast.&lt;&#x2F;p&gt;
&lt;p&gt;This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;optimization-1-hand-rolled-clf-parser&quot;&gt;Optimization 1: Hand-Rolled CLF Parser&lt;a class=&quot;zola-anchor&quot; href=&quot;#optimization-1-hand-rolled-clf-parser&quot; aria-label=&quot;Anchor link for: optimization-1-hand-rolled-clf-parser&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;The CLF timestamp format is completely fixed:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;[DD&#x2F;Mon&#x2F;YYYY:HH:MM:SS ±HHMM]&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt; 0123456789012345678901234 5&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;A hand-written parser only needs to:&lt;&#x2F;p&gt;
&lt;ol&gt;
&lt;li&gt;Find &lt;code&gt;[&lt;&#x2F;code&gt; in the string&lt;&#x2F;li&gt;
&lt;li&gt;Read 6 integers from fixed byte positions&lt;&#x2F;li&gt;
&lt;li&gt;Compute the Unix timestamp via integer arithmetic&lt;&#x2F;li&gt;
&lt;&#x2F;ol&gt;
&lt;p&gt;For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;fn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; days_since_epoch&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; i64&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; m&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; i64&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; d&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; i64&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&amp;gt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; i64&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; m&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;lt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 2&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 1&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; else&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; era&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;=&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 0&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; else&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 399&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 400&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; yoe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; y&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; era&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 400&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; doy&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span&gt; (&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;153&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span&gt; (&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;m&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; m&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;gt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 2&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt;3&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; else&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 9&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 2&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 5&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; d&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 1&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; doe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; yoe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 365&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; yoe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 4&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; yoe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 100&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; doy&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    era&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 146_097&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; doe&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 719_468&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;strong&gt;Result:&lt;&#x2F;strong&gt; parse time dropped from 5.52 s to 1.50 s — &lt;strong&gt;3.7× faster&lt;&#x2F;strong&gt;.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;optimization-2-reuse-the-string-buffer&quot;&gt;Optimization 2: Reuse the String Buffer&lt;a class=&quot;zola-anchor&quot; href=&quot;#optimization-2-reuse-the-string-buffer&quot; aria-label=&quot;Anchor link for: optimization-2-reuse-the-string-buffer&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;The original implementation allocated a new &lt;code&gt;String&lt;&#x2F;code&gt; for every line read:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;&#x2F;&#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; before: fresh allocation per line&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;fn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; read_line&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; dyn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; BufRead&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&amp;gt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Result&lt;&#x2F;span&gt;&lt;span&gt;&amp;lt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Option&lt;&#x2F;span&gt;&lt;span&gt;&amp;lt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;String&lt;&#x2F;span&gt;&lt;span&gt;&amp;gt;&lt;&#x2F;span&gt;&lt;span&gt;&amp;gt;&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;    let&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt; mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; String&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;new&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;  &#x2F;&#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; ← new every time&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;read_line&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;    ...&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;The fix is simple: when an &lt;code&gt;Entry&lt;&#x2F;code&gt; is popped from the heap, you receive &lt;code&gt;line: String&lt;&#x2F;code&gt; with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;&#x2F;&#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; after: capacity is retained&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;while&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt; let&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Some&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Entry&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt; mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt; mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; reader&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; file_idx&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; ..&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; heap&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;pop&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    out&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;write_all&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;line&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;as_bytes&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;    line&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;clear&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;  &#x2F;&#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; ← keeps the heap allocation&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    if&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; fill_line&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; *&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;reader&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;        let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; ts&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; parse_clf_timestamp&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;line&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;unwrap_or_else&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;sentinel&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;        heap&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;push&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;Entry&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; ts&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; file_idx&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; line&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; reader&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no &lt;code&gt;malloc&lt;&#x2F;code&gt; calls in the hot path.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;optimization-3-mimalloc-as-the-global-allocator&quot;&gt;Optimization 3: mimalloc as the Global Allocator&lt;a class=&quot;zola-anchor&quot; href=&quot;#optimization-3-mimalloc-as-the-global-allocator&quot; aria-label=&quot;Anchor link for: optimization-3-mimalloc-as-the-global-allocator&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal &lt;code&gt;BinaryHeap&lt;&#x2F;code&gt; resizing, and the occasional &lt;code&gt;String&lt;&#x2F;code&gt; capacity bump. Swapping out the default glibc allocator for &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;mimalloc&quot;&gt;mimalloc&lt;&#x2F;a&gt; is a one-liner in Rust:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;toml&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;#&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; Cargo.toml&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;mimalloc&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-separator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; version&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-separator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; &amp;quot;&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt;0.1&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-separator&quot;&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; default-features&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-separator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-language&quot;&gt; false&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-punctuation z-definition z-comment z-comment&quot;&gt;&#x2F;&#x2F;&lt;&#x2F;span&gt;&lt;span class=&quot;z-comment&quot;&gt; main.rs&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;use&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; mimalloc&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;MiMalloc&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;#&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span&gt;global_allocator&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage&quot;&gt;static&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt; GLOBAL&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; MiMalloc&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; MiMalloc&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;Result:&lt;&#x2F;strong&gt; 5.55 s → 5.39 s — a further ~3% improvement.&lt;&#x2F;p&gt;
&lt;p&gt;Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;optimization-4-4-mib-read-buffers-simd-newline-search&quot;&gt;Optimization 4: 4 MiB Read Buffers + SIMD Newline Search&lt;a class=&quot;zola-anchor&quot; href=&quot;#optimization-4-4-mib-read-buffers-simd-newline-search&quot; aria-label=&quot;Anchor link for: optimization-4-4-mib-read-buffers-simd-newline-search&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;After the previous three optimizations, profiling showed I&#x2F;O was still the largest remaining cost at 47% of total time. Two changes address this together.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;4 MiB read buffers&lt;&#x2F;strong&gt; — The previous &lt;code&gt;BufReader&lt;&#x2F;code&gt; used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;SIMD newline search via &lt;code&gt;memchr&lt;&#x2F;code&gt;&lt;&#x2F;strong&gt; — Instead of &lt;code&gt;BufReader::read_line&lt;&#x2F;code&gt;, the line reader now uses &lt;code&gt;fill_buf()&lt;&#x2F;code&gt; + &lt;code&gt;memchr::memchr()&lt;&#x2F;code&gt; directly. The &lt;code&gt;memchr&lt;&#x2F;code&gt; crate uses AVX2&#x2F;SSE2 to scan 16 or 32 bytes per cycle when searching for &lt;code&gt;\n&lt;&#x2F;code&gt;:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;rust&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;fn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; fill_line&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; dyn&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; BufRead&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; buf&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;:&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; &amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-storage&quot;&gt;mut&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; String&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; -&amp;gt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Result&lt;&#x2F;span&gt;&lt;span&gt;&amp;lt;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;bool&lt;&#x2F;span&gt;&lt;span&gt;&amp;gt;&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;    loop&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;        let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; available&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;fill_buf&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;        if&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; available&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;is_empty&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword&quot;&gt; return&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Ok&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-language&quot;&gt;false&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;span&gt; }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;        match&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt; memchr&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt;b&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt;&amp;#39;&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-character&quot;&gt;\&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-character&quot;&gt;n&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt;&amp;#39;&lt;&#x2F;span&gt;&lt;span&gt;,&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; available&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;            Some&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;pos&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&amp;gt;&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;                buf&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;push_str&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;std&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;str&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;from_utf8&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;available&lt;&#x2F;span&gt;&lt;span&gt;[&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;..&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;pos&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 1&lt;&#x2F;span&gt;&lt;span&gt;]&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;                reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;consume&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;pos&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; +&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-numeric&quot;&gt; 1&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-keyword&quot;&gt;                return&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; Ok&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-language&quot;&gt;true&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;            }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;            None&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&amp;gt;&lt;&#x2F;span&gt;&lt;span&gt; {&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;                buf&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;push_str&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;std&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;str&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;::&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;from_utf8&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;available&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;?&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-storage z-type z-storage&quot;&gt;                let&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; len&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt; =&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt; available&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;                reader&lt;&#x2F;span&gt;&lt;span class=&quot;z-keyword z-keyword z-operator&quot;&gt;.&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name z-function&quot;&gt;consume&lt;&#x2F;span&gt;&lt;span&gt;(&lt;&#x2F;span&gt;&lt;span class=&quot;z-variable z-other z-variable&quot;&gt;len&lt;&#x2F;span&gt;&lt;span&gt;)&lt;&#x2F;span&gt;&lt;span&gt;;&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;            }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;        }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;    }&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;}&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;strong&gt;Result:&lt;&#x2F;strong&gt; 5.39 s → 4.43 s — &lt;strong&gt;another 18% improvement&lt;&#x2F;strong&gt;, bringing the total to &lt;strong&gt;2.26× faster&lt;&#x2F;strong&gt; than the C original.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;a-road-not-taken-parallel-reader-threads&quot;&gt;A Road Not Taken: Parallel Reader Threads&lt;a class=&quot;zola-anchor&quot; href=&quot;#a-road-not-taken-parallel-reader-threads&quot; aria-label=&quot;Anchor link for: a-road-not-taken-parallel-reader-threads&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;An obvious next step seemed to be spawning one thread per input file — each thread handles its own I&#x2F;O and decompression, sending &lt;code&gt;(timestamp, line)&lt;&#x2F;code&gt; pairs to the merge thread via a bounded &lt;code&gt;mpsc&lt;&#x2F;code&gt; channel:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Thread 1: read + decompress + parse → channel ─┐&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;...                                             ─┘&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;For &lt;strong&gt;compressed files&lt;&#x2F;strong&gt; (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel.&lt;&#x2F;p&gt;
&lt;p&gt;For &lt;strong&gt;plain-text files&lt;&#x2F;strong&gt; on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I&#x2F;O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned &lt;code&gt;String&lt;&#x2F;code&gt; values, making buffer reuse impossible across the thread boundary). The result was slightly &lt;em&gt;slower&lt;&#x2F;em&gt; than the sequential version with large buffers.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;Takeaway:&lt;&#x2F;strong&gt; threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I&#x2F;O resource. Measure first.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;final-results&quot;&gt;Final Results&lt;a class=&quot;zola-anchor&quot; href=&quot;#final-results&quot; aria-label=&quot;Anchor link for: final-results&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines:&lt;&#x2F;p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Version&lt;&#x2F;th&gt;&lt;th&gt;Wall time&lt;&#x2F;th&gt;&lt;th&gt;User&lt;&#x2F;th&gt;&lt;th&gt;System&lt;&#x2F;th&gt;&lt;&#x2F;tr&gt;&lt;&#x2F;thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;mergelog-4.5 (C)&lt;&#x2F;td&gt;&lt;td&gt;10.03 s&lt;&#x2F;td&gt;&lt;td&gt;5.93 s&lt;&#x2F;td&gt;&lt;td&gt;4.12 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;mergelog-rs v1 (jiff)&lt;&#x2F;td&gt;&lt;td&gt;10.39 s&lt;&#x2F;td&gt;&lt;td&gt;9.13 s&lt;&#x2F;td&gt;&lt;td&gt;1.25 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;+ hand-rolled parser&lt;&#x2F;td&gt;&lt;td&gt;5.99 s&lt;&#x2F;td&gt;&lt;td&gt;4.75 s&lt;&#x2F;td&gt;&lt;td&gt;1.23 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;+ buffer reuse&lt;&#x2F;td&gt;&lt;td&gt;5.55 s&lt;&#x2F;td&gt;&lt;td&gt;4.29 s&lt;&#x2F;td&gt;&lt;td&gt;1.26 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;+ mimalloc&lt;&#x2F;td&gt;&lt;td&gt;5.39 s&lt;&#x2F;td&gt;&lt;td&gt;4.17 s&lt;&#x2F;td&gt;&lt;td&gt;1.22 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;+ 4 MiB buffers + memchr&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;strong&gt;4.43 s&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;strong&gt;3.61 s&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;strong&gt;0.81 s&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;&#x2F;tbody&gt;&lt;&#x2F;table&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;memory-and-cpu-usage&quot;&gt;Memory and CPU Usage&lt;a class=&quot;zola-anchor&quot; href=&quot;#memory-and-cpu-usage&quot; aria-label=&quot;Anchor link for: memory-and-cpu-usage&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;Measured with &lt;code&gt;&#x2F;usr&#x2F;bin&#x2F;time -v&lt;&#x2F;code&gt; on the same 7 × 1 GiB workload:&lt;&#x2F;p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;&lt;&#x2F;th&gt;&lt;th&gt;mergelog-4.5&lt;&#x2F;th&gt;&lt;th&gt;mergelog-rs&lt;&#x2F;th&gt;&lt;&#x2F;tr&gt;&lt;&#x2F;thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;Wall time&lt;&#x2F;td&gt;&lt;td&gt;9.63 s&lt;&#x2F;td&gt;&lt;td&gt;4.28 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;User time&lt;&#x2F;td&gt;&lt;td&gt;5.61 s&lt;&#x2F;td&gt;&lt;td&gt;3.42 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;System time&lt;&#x2F;td&gt;&lt;td&gt;4.01 s&lt;&#x2F;td&gt;&lt;td&gt;0.84 s&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;CPU utilisation&lt;&#x2F;td&gt;&lt;td&gt;99%&lt;&#x2F;td&gt;&lt;td&gt;99%&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;strong&gt;Peak RSS&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;strong&gt;1.6 MiB&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;strong&gt;30.7 MiB&lt;&#x2F;strong&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;Involuntary ctx switches&lt;&#x2F;td&gt;&lt;td&gt;105&lt;&#x2F;td&gt;&lt;td&gt;122&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;&#x2F;tbody&gt;&lt;&#x2F;table&gt;
&lt;p&gt;&lt;strong&gt;Memory&lt;&#x2F;strong&gt; — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write &lt;code&gt;BufWriter&lt;&#x2F;code&gt;, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in &lt;code&gt;reader.rs&lt;&#x2F;code&gt; that can be tuned if memory is constrained.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;System time&lt;&#x2F;strong&gt; — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls &lt;code&gt;write(1, ...)&lt;&#x2F;code&gt; directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via &lt;code&gt;BufWriter&lt;&#x2F;code&gt; and reads via 4 MiB &lt;code&gt;BufReader&lt;&#x2F;code&gt;, cutting kernel round-trips dramatically.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;CPU&lt;&#x2F;strong&gt; — Both tools are single-threaded and max out one core at 99%.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;takeaways&quot;&gt;Takeaways&lt;a class=&quot;zola-anchor&quot; href=&quot;#takeaways&quot; aria-label=&quot;Anchor link for: takeaways&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;Five lessons from this project:&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;1. Measure before optimizing.&lt;&#x2F;strong&gt; Without the timing binary the focus might have landed on the heap. But &lt;code&gt;jiff::strptime&lt;&#x2F;code&gt; was costing nearly 3× more than every other phase combined.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;2. Libraries are great until they aren’t.&lt;&#x2F;strong&gt; &lt;code&gt;jiff&lt;&#x2F;code&gt; is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;3. Sometimes the allocator is the last few percent.&lt;&#x2F;strong&gt; After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;4. Buffer size matters more than you think.&lt;&#x2F;strong&gt; Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;5. Parallelism requires the right bottleneck.&lt;&#x2F;strong&gt; Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I&#x2F;O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly &lt;em&gt;slower&lt;&#x2F;em&gt;. Threading is a tool, not a universal solution.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;6. Ownership is not a burden.&lt;&#x2F;strong&gt; Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — &lt;code&gt;line.clear()&lt;&#x2F;code&gt; instead of &lt;code&gt;free(line); line = malloc(...)&lt;&#x2F;code&gt;, with the compiler guaranteeing no use-after-free.&lt;&#x2F;p&gt;
&lt;p&gt;The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this.&lt;&#x2F;p&gt;
</content>
        
    </entry>
    <entry xml:lang="en">
        <title>AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</title>
        <published>2026-03-12T00:00:00+00:00</published>
        <updated>2026-03-12T00:00:00+00:00</updated>
        
        <author>
          <name>
            
              aleks
            
          </name>
        </author>
        
        <link rel="alternate" type="text/html" href="https://blog.none.at/blog/2026/2026-03-12-blog-awffull/"/>
        <id>https://blog.none.at/blog/2026/2026-03-12-blog-awffull/</id>
        
        <content type="html" xml:base="https://blog.none.at/blog/2026/2026-03-12-blog-awffull/">&lt;h1 id=&quot;awffull-4-0-0-modernising-a-2008-web-log-analyser&quot;&gt;AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser&lt;a class=&quot;zola-anchor&quot; href=&quot;#awffull-4-0-0-modernising-a-2008-web-log-analyser&quot; aria-label=&quot;Anchor link for: awffull-4-0-0-modernising-a-2008-web-log-analyser&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h1&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;background&quot;&gt;Background&lt;a class=&quot;zola-anchor&quot; href=&quot;#background&quot; aria-label=&quot;Anchor link for: background&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;&lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;gitlab.com&#x2F;aleks001&#x2F;awfull&quot;&gt;AWFFull 4.0.0&lt;&#x2F;a&gt; is a web server log analyser. It
reads Apache&#x2F;nginx&#x2F;Squid access logs and produces HTML reports with graphs
showing traffic, top URLs, referrers, user agents, country statistics, and
more. It was originally forked from
&lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;www.mrunix.net&#x2F;webalizer&#x2F;&quot;&gt;The Webalizer&lt;&#x2F;a&gt; by Bradford L. Barrett
(1997–2001), extended as AWFFull by Steve McInerney from 2004, and actively
developed until 2008, when it was last released as version 3.10.2.&lt;&#x2F;p&gt;
&lt;p&gt;It still works well in 2026 — the core log-parsing logic is solid and the
output is clean. But the codebase had accumulated 18 years of code: EOL
libraries, dead links, obsolete command-line flags, a broken GeoIP
implementation, and a very old &lt;code&gt;config.guess&lt;&#x2F;code&gt;. Time for a
proper cleanup.&lt;&#x2F;p&gt;
&lt;p&gt;This post covers what changed in AWFFull 4.0.0 and why.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;is-a-static-web-log-analyser-still-useful&quot;&gt;Is a Static Web Log Analyser Still Useful?&lt;a class=&quot;zola-anchor&quot; href=&quot;#is-a-static-web-log-analyser-still-useful&quot; aria-label=&quot;Anchor link for: is-a-static-web-log-analyser-still-useful&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;Good question. The short answer is: yes — and arguably more so than before.&lt;&#x2F;p&gt;
&lt;p&gt;Modern web analytics platforms — Google Analytics, Matomo, Plausible, Grafana
with Loki, ELK stacks — are excellent for live dashboards and drill-down
exploration. But they share a common weakness: &lt;strong&gt;they are slow when you need to
process large volumes of historical log data and render the results as a
complete set of graphs and tables&lt;&#x2F;strong&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;Feed a year of Apache logs (say, 500 million lines across 50 GB of gzipped
files) into an ELK pipeline and you’ll be waiting a long time for the data to
be ingested, indexed, and ready to query. Then wait again each time you open a
dashboard. Grafana rendering a year’s worth of nginx traffic across 30 panels?
Expect seconds of load time per page, per query.&lt;&#x2F;p&gt;
&lt;p&gt;AWFFull’s approach is different: it processes the raw log files directly, keeps
all state in a compact binary history file, and writes &lt;strong&gt;static HTML + PNG
output&lt;&#x2F;strong&gt; that loads instantly in any browser — even on a machine from 2010,
over a slow connection, or offline entirely. There is no query layer, no index,
no database, no running server. The report is just files.&lt;&#x2F;p&gt;
&lt;p&gt;For the typical use case — a sysadmin or developer who wants a monthly
overview of who accessed their server, from where, with what agents, hitting
which URLs — this is exactly the right trade-off:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Fast to generate&lt;&#x2F;strong&gt;: processing a month of logs takes seconds, not minutes&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Instant to view&lt;&#x2F;strong&gt;: static HTML, no round-trips, no query engine&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Self-contained&lt;&#x2F;strong&gt;: the report directory can be archived, moved, or served
from any static file server&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Privacy-friendly&lt;&#x2F;strong&gt;: no third-party JavaScript, no tracking pixels, no data
leaving your infrastructure — the raw IP addresses stay in your own logs&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Works on air-gapped systems&lt;&#x2F;strong&gt;: nothing requires internet access at runtime&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;It also complements rather than replaces live monitoring. You would typically
use Prometheus&#x2F;Grafana (or similar) to watch your infrastructure in real time,
and AWFFull to produce the monthly historical report that goes into a ticket or
a management summary.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;shipping-access-logs-to-object-storage-and-running-in-containers&quot;&gt;Shipping Access Logs to Object Storage and Running in Containers&lt;a class=&quot;zola-anchor&quot; href=&quot;#shipping-access-logs-to-object-storage-and-running-in-containers&quot; aria-label=&quot;Anchor link for: shipping-access-logs-to-object-storage-and-running-in-containers&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The operational details — how to ship access logs to S3 or Azure Blob with
Fluent Bit, Vector, or Filebeat&#x2F;Logstash, how to handle concurrent writes and
log loss at high traffic, and how to run AWFFull as a stateless container
against object storage — are covered in the companion post:&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;&lt;a href=&quot;https:&#x2F;&#x2F;blog.none.at&#x2F;blog&#x2F;2026&#x2F;2026-03-12-blog-log-shipping&#x2F;&quot;&gt;AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers&lt;&#x2F;a&gt;&lt;&#x2F;strong&gt;&lt;&#x2F;p&gt;
&lt;h2 id=&quot;the-big-ticket-items&quot;&gt;The Big Ticket Items&lt;a class=&quot;zola-anchor&quot; href=&quot;#the-big-ticket-items&quot; aria-label=&quot;Anchor link for: the-big-ticket-items&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;h3 id=&quot;pcre-pcre2&quot;&gt;PCRE → PCRE2&lt;a class=&quot;zola-anchor&quot; href=&quot;#pcre-pcre2&quot; aria-label=&quot;Anchor link for: pcre-pcre2&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;AWFFull used libpcre (PCRE version 1) for log-line regex parsing. PCRE 1 has
been officially end-of-life since 2012 and is being dropped by distributions.
We migrated to &lt;strong&gt;PCRE2&lt;&#x2F;strong&gt;, the current maintained successor with a cleaner API
and better Unicode support. The change touched &lt;code&gt;parser.c&lt;&#x2F;code&gt;, &lt;code&gt;common.h&lt;&#x2F;code&gt;, and
&lt;code&gt;configure.ac&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;geoip-from-legacy-dat-to-libmaxminddb&quot;&gt;GeoIP: From Legacy &lt;code&gt;.dat&lt;&#x2F;code&gt; to libmaxminddb&lt;a class=&quot;zola-anchor&quot; href=&quot;#geoip-from-legacy-dat-to-libmaxminddb&quot; aria-label=&quot;Anchor link for: geoip-from-legacy-dat-to-libmaxminddb&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The old code used &lt;strong&gt;libGeoIP&lt;&#x2F;strong&gt; with a proprietary binary &lt;code&gt;.dat&lt;&#x2F;code&gt; format that
MaxMind deprecated in 2019. Country lookups were broken for anyone who hadn’t
kept a seven-year-old database around.&lt;&#x2F;p&gt;
&lt;p&gt;AWFFull 4.0.0 replaces this with &lt;strong&gt;libmaxminddb&lt;&#x2F;strong&gt;, the open standard MMDB
format used by both MaxMind (GeoLite2) and DB-IP. The migration is
transparent — IPv4 and IPv6 are both handled by a single &lt;code&gt;MMDB_lookup_string()&lt;&#x2F;code&gt;
call, with no separate code paths needed.&lt;&#x2F;p&gt;
&lt;p&gt;Free databases updated monthly:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Country&lt;&#x2F;strong&gt;: &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;db-ip.com&#x2F;db&#x2F;download&#x2F;ip-to-country-lite&quot;&gt;DB-IP Country Lite&lt;&#x2F;a&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;ASN&lt;&#x2F;strong&gt;: &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;db-ip.com&#x2F;db&#x2F;download&#x2F;ip-to-asn-lite&quot;&gt;DB-IP ASN Lite&lt;&#x2F;a&gt;&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;h3 id=&quot;asn-statistics-new-table-and-pie-chart&quot;&gt;ASN Statistics: New Table and Pie Chart&lt;a class=&quot;zola-anchor&quot; href=&quot;#asn-statistics-new-table-and-pie-chart&quot; aria-label=&quot;Anchor link for: asn-statistics-new-table-and-pie-chart&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;Once you have a MMDB-capable lookup layer, adding ASN (Autonomous System
Number) statistics is almost free. A second database (&lt;code&gt;GeoASNDatabase&lt;&#x2F;code&gt;) maps
IPs to their network operator — think “Cloudflare”, “Amazon AWS”,
“Deutsche Telekom”. The report now shows a &lt;strong&gt;Top ASNs&lt;&#x2F;strong&gt; table and pie chart,
answering the question &lt;em&gt;“which networks send the most traffic to my site?”&lt;&#x2F;em&gt;&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;GeoASN          yes&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;GeoASNDatabase  &#x2F;usr&#x2F;share&#x2F;GeoIP&#x2F;dbip-asn-lite.mmdb&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Performance note: both GeoIP and GeoASN lookups happen in the &lt;em&gt;output phase&lt;&#x2F;em&gt;,
once per unique IP address — not once per log line. On a typical site the
overhead is negligible.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;hyperscan-simd-pattern-matching&quot;&gt;Hyperscan: SIMD Pattern Matching&lt;a class=&quot;zola-anchor&quot; href=&quot;#hyperscan-simd-pattern-matching&quot; aria-label=&quot;Anchor link for: hyperscan-simd-pattern-matching&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;AWFFull supports extensive pattern lists for ignoring, grouping, and hiding
hosts, URLs, referrers, and agents. The original code used a
&lt;strong&gt;Boyer-Moore-Horspool&lt;&#x2F;strong&gt; linear scan: each log record was matched against every
pattern in turn.&lt;&#x2F;p&gt;
&lt;p&gt;When &lt;strong&gt;Intel Hyperscan&lt;&#x2F;strong&gt; (&lt;code&gt;libhyperscan&lt;&#x2F;code&gt;) is installed, AWFFull now compiles all
patterns from a given list into a single Hyperscan database and matches them in
one SIMD pass per record. For large pattern lists — say, 50+ bot&#x2F;spider agent
patterns — this is significantly faster. Hyperscan is auto-detected at
configure time and falls back to the existing BMH path if unavailable.&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;sudo&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; apt&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; install&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; libhyperscan-dev&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;.&#x2F;configure&lt;&#x2F;span&gt;&lt;span class=&quot;z-punctuation z-separator&quot;&gt; &amp;amp;&amp;amp;&lt;&#x2F;span&gt;&lt;span class=&quot;z-entity z-name&quot;&gt; make&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;&lt;h3 id=&quot;mimalloc-better-heap-performance&quot;&gt;mimalloc: Better Heap Performance&lt;a class=&quot;zola-anchor&quot; href=&quot;#mimalloc-better-heap-performance&quot; aria-label=&quot;Anchor link for: mimalloc-better-heap-performance&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;For large log files with millions of unique hosts, URLs, and referrers, heap
allocation overhead becomes measurable. Adding &lt;strong&gt;mimalloc&lt;&#x2F;strong&gt; (Microsoft’s
high-performance allocator) as an optional drop-in was straightforward via the
explicit &lt;code&gt;mi_calloc&lt;&#x2F;code&gt; API. Auto-detected at configure time.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;xxh3-hash-function&quot;&gt;XXH3 Hash Function&lt;a class=&quot;zola-anchor&quot; href=&quot;#xxh3-hash-function&quot; aria-label=&quot;Anchor link for: xxh3-hash-function&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The internal hash table used djb2, a good general-purpose hash from 1991.
AWFFull 4.0.0 replaces it with &lt;strong&gt;XXH3&lt;&#x2F;strong&gt; (via the bundled &lt;code&gt;xxhash.h&lt;&#x2F;code&gt;), which
offers better distribution and significantly higher throughput on modern CPUs.
The table load-factor was also tuned to reduce collisions.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;user-visible-improvements&quot;&gt;User-Visible Improvements&lt;a class=&quot;zola-anchor&quot; href=&quot;#user-visible-improvements&quot; aria-label=&quot;Anchor link for: user-visible-improvements&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;h3 id=&quot;date-stamp-on-graphs&quot;&gt;Date Stamp on Graphs&lt;a class=&quot;zola-anchor&quot; href=&quot;#date-stamp-on-graphs&quot; aria-label=&quot;Anchor link for: date-stamp-on-graphs&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;Every generated PNG graph now shows the &lt;strong&gt;current date&lt;&#x2F;strong&gt; (&lt;code&gt;YYYY-MM-DD&lt;&#x2F;code&gt;) in the
top-right corner, aligned with the title baseline. Small but useful when you’re
looking at archived reports six months later.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;next-previous-month-navigation&quot;&gt;Next &#x2F; Previous Month Navigation&lt;a class=&quot;zola-anchor&quot; href=&quot;#next-previous-month-navigation&quot; aria-label=&quot;Anchor link for: next-previous-month-navigation&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;Monthly report pages now show navigation links at the top:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;plain&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span&gt;← February 2026  |  [Index]  |  April 2026 →&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Links are derived from the history file — only months that actually exist in
the database are linked.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;updated-browser-bot-detection&quot;&gt;Updated Browser&#x2F;Bot Detection&lt;a class=&quot;zola-anchor&quot; href=&quot;#updated-browser-bot-detection&quot; aria-label=&quot;Anchor link for: updated-browser-bot-detection&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The &lt;code&gt;sample.conf&lt;&#x2F;code&gt; browser&#x2F;agent detection list dated from 2005 and contained
entries like “Netscape 4.5”, “Internet Explorer 3.0 (Win95)”, and
“MSNBot” — none of which have been seen in the wild for a decade or more. The
list has been completely rewritten:&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;Removed:&lt;&#x2F;strong&gt; Netscape 1–4.x, Internet Explorer 3–7, Firebird, Galeon, Camino,
msnbot (all entries)&lt;&#x2F;p&gt;
&lt;p&gt;&lt;strong&gt;Added:&lt;&#x2F;strong&gt; Chrome&#x2F;Chromium, Edge (Chromium), Brave, Samsung Browser, Firefox,
Safari, Opera (OPR&#x2F;), modern search bots (Bingbot, DuckDuckBot, YandexBot,
Baidu, Applebot, AhrefsBot, SemrushBot), AI crawlers (GPTBot, ClaudeBot,
PerplexityBot, Bytespider, CCBot, Amazonbot), and CLI tools (curl, Wget,
python-requests).&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;build-system-and-tooling&quot;&gt;Build System and Tooling&lt;a class=&quot;zola-anchor&quot; href=&quot;#build-system-and-tooling&quot; aria-label=&quot;Anchor link for: build-system-and-tooling&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;h3 id=&quot;cli-cleanup-14-flags-removed&quot;&gt;CLI Cleanup: 14 Flags Removed&lt;a class=&quot;zola-anchor&quot; href=&quot;#cli-cleanup-14-flags-removed&quot; aria-label=&quot;Anchor link for: cli-cleanup-14-flags-removed&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;AWFFull inherited many single-letter flags from the original Webalizer. Flags
like &lt;code&gt;-G&lt;&#x2F;code&gt; (suppress hourly graph), &lt;code&gt;-L&lt;&#x2F;code&gt; (suppress legend), &lt;code&gt;-M&lt;&#x2F;code&gt; (mangle
agents), &lt;code&gt;-x&lt;&#x2F;code&gt; (HTML extension), and &lt;code&gt;-I&lt;&#x2F;code&gt; (index alias) had exact equivalents
in the config file. At version 4.0.0 — a major version bump — these were
removed from the CLI, along with three fully deprecated no-op flags (&lt;code&gt;-d&lt;&#x2F;code&gt;,
&lt;code&gt;-q&lt;&#x2F;code&gt;, &lt;code&gt;-Q&lt;&#x2F;code&gt;).&lt;&#x2F;p&gt;
&lt;p&gt;The &lt;code&gt;--help&lt;&#x2F;code&gt; output now documents the new verbosity levels (&lt;code&gt;-v&lt;&#x2F;code&gt; through
&lt;code&gt;-vvvvv&lt;&#x2F;code&gt;) and includes a note pointing config-file-only settings to
&lt;code&gt;awffull.conf(5)&lt;&#x2F;code&gt;.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;man-pages-docbook-xml-pandoc-markdown&quot;&gt;Man Pages: DocBook XML → Pandoc Markdown&lt;a class=&quot;zola-anchor&quot; href=&quot;#man-pages-docbook-xml-pandoc-markdown&quot; aria-label=&quot;Anchor link for: man-pages-docbook-xml-pandoc-markdown&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;The existing man pages were written in DocBook XML 4.5 and built with
&lt;code&gt;xsltproc&lt;&#x2F;code&gt; — a tool that requires &lt;code&gt;xsltproc&lt;&#x2F;code&gt;, &lt;code&gt;docbook-xsl&lt;&#x2F;code&gt;, and several XML
schema packages, none of which are installed by default on most systems in
2026.&lt;&#x2F;p&gt;
&lt;p&gt;The man pages are now maintained as &lt;strong&gt;pandoc Markdown&lt;&#x2F;strong&gt; (&lt;code&gt;doc&#x2F;awffull.1.md&lt;&#x2F;code&gt;,
&lt;code&gt;doc&#x2F;awffull.conf.5.md&lt;&#x2F;code&gt;) and built with:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;pandoc&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt;s&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt;t&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; man&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; awffull.1.md&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt;o&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; awffull.1&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;code&gt;configure.ac&lt;&#x2F;code&gt; auto-detects pandoc. Pre-built man pages remain in the
repository as a fallback for systems without pandoc. The DocBook XML sources
are retained in &lt;code&gt;doc&#x2F;&lt;&#x2F;code&gt; for historical reference.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;translations-quality-over-quantity&quot;&gt;Translations: Quality Over Quantity&lt;a class=&quot;zola-anchor&quot; href=&quot;#translations-quality-over-quantity&quot; aria-label=&quot;Anchor link for: translations-quality-over-quantity&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h3&gt;
&lt;p&gt;AWFFull shipped with 33 language files, most of which were approximately 29% complete and
had not been updated since 2008. Rather than shipping incomplete translations
that silently fall back to English mid-report, we reduced to &lt;strong&gt;6 fully
complete languages&lt;&#x2F;strong&gt; (German, Finnish, Italian, Swedish, Brazilian Portuguese,
Indonesian) and brought all of them to &lt;strong&gt;100% (524&#x2F;524 strings)&lt;&#x2F;strong&gt;.&lt;&#x2F;p&gt;
&lt;p&gt;The &lt;code&gt;.pot&lt;&#x2F;code&gt; template was regenerated with &lt;code&gt;xgettext&lt;&#x2F;code&gt; to include all new strings
added in 4.0.0 (GeoASN messages, MaxMind DB errors, pattern matching warnings).&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;repository-modernisation&quot;&gt;Repository Modernisation&lt;a class=&quot;zola-anchor&quot; href=&quot;#repository-modernisation&quot; aria-label=&quot;Anchor link for: repository-modernisation&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;The project moved from a collection of legacy files to a cleaner structure:&lt;&#x2F;p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Before&lt;&#x2F;th&gt;&lt;th&gt;After&lt;&#x2F;th&gt;&lt;&#x2F;tr&gt;&lt;&#x2F;thead&gt;&lt;tbody&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;README&lt;&#x2F;code&gt; + &lt;code&gt;README.FIRST&lt;&#x2F;code&gt; + &lt;code&gt;README.webalizer&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;README.md&lt;&#x2F;code&gt; (single entry point)&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;ChangeLog&lt;&#x2F;code&gt; (GNU format, frozen)&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;CHANGELOG.md&lt;&#x2F;code&gt; (Keep a Changelog)&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;COPYING&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;LICENSE&lt;&#x2F;code&gt;&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;TODO&lt;&#x2F;code&gt; (plain text, 2008)&lt;&#x2F;td&gt;&lt;td&gt;&lt;code&gt;TODO.md&lt;&#x2F;code&gt; (Markdown with &lt;code&gt;[x]&lt;&#x2F;code&gt; checkboxes)&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;country-codes.txt&lt;&#x2F;code&gt; (2005, informational)&lt;&#x2F;td&gt;&lt;td&gt;removed&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;DNS.README&lt;&#x2F;code&gt; (dead link)&lt;&#x2F;td&gt;&lt;td&gt;removed&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;contrib&#x2F;&lt;&#x2F;code&gt; (Webalizer migration scripts)&lt;&#x2F;td&gt;&lt;td&gt;removed&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;tr&gt;&lt;td&gt;&lt;code&gt;config.guess&lt;&#x2F;code&gt; from 2008&lt;&#x2F;td&gt;&lt;td&gt;updated to automake 1.16 (2022)&lt;&#x2F;td&gt;&lt;&#x2F;tr&gt;
&lt;&#x2F;tbody&gt;&lt;&#x2F;table&gt;

&lt;link rel=&quot;stylesheet&quot; href=&quot;https:&#x2F;&#x2F;blog.none.at&#x2F;admonition.css?h=98c1477488c4b2c9d71a&quot; type=&quot;text&#x2F;css&quot;&gt;

&lt;div
  class=&quot;my-4 flex flex-col rounded-lg bg-(--admonition-bg)&quot;
  style=&quot;--admonition-bg: var(--admonition-tip);&quot;
&gt;
  &lt;div class=&quot;flex items-center rounded-t-lg bg-(--admonition-bg) p-1&quot;&gt;
    &lt;div
      class=&quot;mx-2 h-4 w-4 text-[0px] [background:var(--url)_center_center_no-repeat] dark:invert&quot;
      style=&quot;--url: url(icons&#x2F;tip.svg);&quot;
    &gt;
      tip
    &lt;&#x2F;div&gt;
    &lt;span&gt;&lt;strong&gt;tip&lt;&#x2F;strong&gt;&lt;&#x2F;span&gt;
  &lt;&#x2F;div&gt;
  &lt;div class=&quot;pl-4&quot;&gt;&lt;p&gt;The full list of technical changes is available in &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;gitlab.com&#x2F;aleks001&#x2F;awffull&#x2F;-&#x2F;blob&#x2F;main&#x2F;CHANGES.md&quot;&gt;CHANGES.md&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;&#x2F;div&gt;
&lt;&#x2F;div&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;ci-cd-pipeline&quot;&gt;CI&#x2F;CD Pipeline&lt;a class=&quot;zola-anchor&quot; href=&quot;#ci-cd-pipeline&quot; aria-label=&quot;Anchor link for: ci-cd-pipeline&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;A full GitLab CI pipeline was added covering:&lt;&#x2F;p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Build verification&lt;&#x2F;strong&gt; on Debian 12&#x2F;13, Ubuntu 22.04&#x2F;24.04, AlmaLinux 8&#x2F;9&#x2F;10&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Source and binary tarballs&lt;&#x2F;strong&gt; on every push to &lt;code&gt;main&lt;&#x2F;code&gt;&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;.deb&lt;&#x2F;code&gt; package&lt;&#x2F;strong&gt; (Ubuntu 24.04 amd64) on release tags&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;.rpm&lt;&#x2F;code&gt; package&lt;&#x2F;strong&gt; (AlmaLinux 9 amd64) on release tags&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;Container image&lt;&#x2F;strong&gt; pushed to GitLab Container Registry (Ubuntu 24.04 base, all optional libs included)&lt;&#x2F;li&gt;
&lt;li&gt;&lt;strong&gt;GitLab Release&lt;&#x2F;strong&gt; creation with all artifacts uploaded to the Generic Package Registry&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;Releases are triggered by pushing a semver tag:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;git&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; tag&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; v4.0.0&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;git&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; push&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt; -&lt;&#x2F;span&gt;&lt;span class=&quot;z-constant z-constant z-other&quot;&gt;-tags&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;The container image is available as:&lt;&#x2F;p&gt;
&lt;pre class=&quot;giallo z-code&quot;&gt;&lt;code data-lang=&quot;shellscript&quot;&gt;&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;docker&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; pull&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; registry.gitlab.com&#x2F;aleks001&#x2F;awffull:v4.0.0&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;
&lt;span class=&quot;giallo-l&quot;&gt;&lt;span class=&quot;z-entity z-name&quot;&gt;docker&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; pull&lt;&#x2F;span&gt;&lt;span class=&quot;z-string&quot;&gt; registry.gitlab.com&#x2F;aleks001&#x2F;awffull:latest&lt;&#x2F;span&gt;&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;The other artifacts are available on the &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;gitlab.com&#x2F;aleks001&#x2F;awffull&#x2F;-&#x2F;releases&quot;&gt;Release Page&lt;&#x2F;a&gt;&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;h2 id=&quot;what-s-next&quot;&gt;What’s Next&lt;a class=&quot;zola-anchor&quot; href=&quot;#what-s-next&quot; aria-label=&quot;Anchor link for: what-s-next&quot;&gt;🔗&lt;&#x2F;a&gt;&lt;&#x2F;h2&gt;
&lt;p&gt;The &lt;code&gt;TODO.md&lt;&#x2F;code&gt; tracks longer-term ideas: SVG graphs (replacing the aging
libgd dependency), weekly summaries, a templating system for the HTML output,
and better browser&#x2F;OS reporting. Contributions welcome.&lt;&#x2F;p&gt;
&lt;p&gt;The project is at &lt;a rel=&quot;noopener external&quot; target=&quot;_blank&quot; href=&quot;https:&#x2F;&#x2F;gitlab.com&#x2F;aleks001&#x2F;awfull&quot;&gt;https:&#x2F;&#x2F;gitlab.com&#x2F;aleks001&#x2F;awfull&lt;&#x2F;a&gt;.&lt;&#x2F;p&gt;
&lt;hr &#x2F;&gt;
&lt;p&gt;&lt;em&gt;AWFFull is free software licensed under GPL v3 or later.&lt;&#x2F;em&gt;
&lt;em&gt;Based on The Webalizer by Bradford L. Barrett (1997–2001).&lt;&#x2F;em&gt;
&lt;em&gt;Extended by Steve McInerney (2004–2008) and Aleksandar Lazic (2026–present).&lt;&#x2F;em&gt;&lt;&#x2F;p&gt;
</content>
        
    </entry>
</feed>
