mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster

aleks — Sun, 15 Mar 2026 00:00:00 +0000

mergelog</a> is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. The original version 4.5 was written in C around 2000–2001, weighs in at ~450 lines, and does exactly what it promises — reliably and fast.

This post documents the journey from mergelog-4.5 to mergelog-rs</a>: what changed, what was surprising, and how six targeted optimizations made the Rust binary 2.26× faster than the C original.

Full respect and honour to Bertrand Demiddelaer, who created mergelog back in 2000. Writing a tool this fast, this correct, and this compact in C — with no external dependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his shoulders.

The Original: mergelog 4.5 🔗</a></h2>
The C program reads any number of log files in Common Log Format (CLF) / NCSA Combined Format:
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"</code></pre> Example line:93.184.216.34 - - [15/Mar/2026:12:00:00 +0100] "GET /index.html HTTP/1.1" 200 1234 "-" "curl/7.88"</code></pre> The algorithm in 4.5 is straightforward: find the earliest timestamp across all files, then iterate second-by-second from there to the end, writing all matching lines at each step. That is O(timespan × K) — slow when logs span multiple years or many files are involved. Gzip support exists, but as a separate binary (zmergelog</code>) compiled with -DUSE_ZLIB</code>. The Rust Rewrite: What Changed🔗</a></h2> 1. Algorithm: K-Way Merge with a Binary Heap🔗</a></h3> Instead of time-based iteration, mergelog-rs uses a k-way merge with a min-heap (std::collections::BinaryHeap</code> with reversed ordering): Seed the heap with the first line from each file</li> Main loop: pop the smallest timestamp, write the line, push the next line from the same file</li> Complexity: O(N log K) — independent of the time span of the logs</li> </ul> while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre>2. Magic-Byte Detection Instead of File Extension🔗</a></h3> The original detects gzip by binary type (zmergelog</code> vs. mergelog</code>). mergelog-rs reads the first 6 bytes of each file and detects the format automatically: Format</th> Magic Bytes</th></tr></thead> gzip</td> 1F 8B</code></td></tr> bzip2</td> 42 5A 68</code> (BZh</code>)</td></tr> xz</td> FD 37 7A 58 5A 00</code></td></tr> zstd</td> 28 B5 2F FD</code></td></tr> </tbody></table> pub fn detect_from_bytes(magic: &[u8]) -> Compression { if magic.len() >= 2 && magic[..2] == [0x1f, 0x8b] { Compression::Gz } else if magic.len() >= 3 && magic[..3] == [0x42, 0x5a, 0x68] { Compression::Bz2 } else if magic.len() >= 6 && magic[..6] == [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00] { Compression::Xz } else if magic.len() >= 4 && magic[..4] == [0x28, 0xb5, 0x2f, 0xfd] { Compression::Zstd } else { Compression::None } }</code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed. 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output. mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone. Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>: ──── profile_timing (41.65M lines) ────────────────── I/O (read_line) 3.91 s 32.4% Parse (jiff strptime) 5.52 s 45.7% ← bottleneck Heap (push/pop) 1.97 s 16.3% Write (write_all) 0.69 s 5.7% ───────────────────────────────────────────────── Σ 12.08 s</code></pre> Nearly half the time was in the parser. The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast. This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser. Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed: [DD/Mon/YYYY:HH:MM:SS ±HHMM] 0123456789012345678901234 5</code></pre> A hand-written parser only needs to: Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication: fn days_since_epoch(y: i64, m: i64, d: i64) -> i64 { let y = if m <= 2 { y - 1 } else { y }; let era = if y >= 0 { y } else { y - 399 } / 400; let yoe = y - era * 400; let doy = (153 * (m + if m > 2 { -3 } else { 9 }) + 2) / 5 + d - 1; let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; era * 146_097 + doe - 719_468 }</code></pre> Result: parse time dropped from 5.52 s to 1.50 s — 3.7× faster. Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read: // before: fresh allocation per line fn read_line(reader: &mut dyn BufRead) -> Result<Option<String>> { let mut line = String::new(); // ← new every time reader.read_line(&mut line)?; ... }</code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer: // after: capacity is retained while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); // ← keeps the heap allocation if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path. Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust: # Cargo.toml mimalloc = { version = "0.1", default-features = false }</code></pre>// main.rs use mimalloc::MiMalloc; #[global_allocator] static GLOBAL: MiMalloc = MiMalloc;</code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well. Result: 5.55 s → 5.39 s — a further ~3% improvement. Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else. Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together. 4 MiB read buffers — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time. SIMD newline search via memchr</code> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>: fn fill_line(reader: &mut dyn BufRead, buf: &mut String) -> Result<bool> { loop { let available = reader.fill_buf()?; if available.is_empty() { return Ok(false); } match memchr(b'\n', available) { Some(pos) => { buf.push_str(std::str::from_utf8(&available[..pos + 1])?); reader.consume(pos + 1); return Ok(true); } None => { buf.push_str(std::str::from_utf8(available)?); let len = available.len(); reader.consume(len); } } } }</code></pre> Result: 5.39 s → 4.43 s — another 18% improvement, bringing the total to 2.26× faster than the C original. A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel: Thread 1: read + decompress + parse → channel ─┐ Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout ... ─┘</code></pre> For compressed files (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel. For plain-text files on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower than the sequential version with large buffers. Takeaway: threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first. Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines: Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</td> 4.43 s</td> 3.61 s</td> 0.81 s</td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload: </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</td> 1.6 MiB</td> 30.7 MiB</td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained. System time — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically. CPU — Both tools are single-threaded and max out one core at 99%. Takeaways🔗</a></h2> Five lessons from this project: 1. Measure before optimizing. Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined. 2. Libraries are great until they aren’t. jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×. 3. Sometimes the allocator is the last few percent. After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying. 4. Buffer size matters more than you think. Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better. 5. Parallelism requires the right bottleneck. Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower. Threading is a tool, not a universal solution. 6. Ownership is not a burden. Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free. The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this.

Format</th>	Magic Bytes</th></tr></thead>
gzip</td>	`1F 8B</code></td></tr>`
bzip2</td>	`42 5A 68</code> (BZh</code>)</td></tr>`
xz</td>	`FD 37 7A 58 5A 00</code></td></tr>`
zstd</td>	28 B5 2F FD</code></td></tr> </tbody></table> pub</span> fn</span> detect_from_bytes</span>(</span>magic</span>:</span> &</span>[</span>u8</span>]</span>)</span> -></span> Compression</span> {</span></span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 2</span> &&</span> magic</span>[</span>..</span>2</span>]</span> ==</span> [</span>0x1f</span>,</span> 0x8b</span>]</span> {</span> Compression</span>::</span>Gz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 3</span> &&</span> magic</span>[</span>..</span>3</span>]</span> ==</span> [</span>0x42</span>,</span> 0x5a</span>,</span> 0x68</span>]</span> {</span> Compression</span>::</span>Bz2</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 6</span> &&</span> magic</span>[</span>..</span>6</span>]</span> ==</span> [</span>0xfd</span>,</span> 0x37</span>,</span> 0x7a</span>,</span> 0x58</span>,</span> 0x5a</span>,</span> 0x00</span>]</span> {</span> Compression</span>::</span>Xz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 4</span> &&</span> magic</span>[</span>..</span>4</span>]</span> ==</span> [</span>0x28</span>,</span> 0xb5</span>,</span> 0x2f</span>,</span> 0xfd</span>]</span> {</span> Compression</span>::</span>Zstd</span> }</span></span> else</span> {</span> Compression</span>::</span>None</span> }</span></span> }</span></span></code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed.</p> 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output.</p> mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone.</p> Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>:</p> ──── profile_timing (41.65M lines) ──────────────────</span></span> I/O (read_line) 3.91 s 32.4%</span></span> Parse (jiff strptime) 5.52 s 45.7% ← bottleneck</span></span> Heap (push/pop) 1.97 s 16.3%</span></span> Write (write_all) 0.69 s 5.7%</span></span> ─────────────────────────────────────────────────</span></span> Σ 12.08 s</span></span></code></pre> Nearly half the time was in the parser.</strong> The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast.</p> This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser.</p> Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed:</p> [DD/Mon/YYYY:HH:MM:SS ±HHMM]</span></span> 0123456789012345678901234 5</span></span></code></pre> A hand-written parser only needs to:</p> Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication:</p> fn</span> days_since_epoch</span>(</span>y</span>:</span> i64</span>,</span> m</span>:</span> i64</span>,</span> d</span>:</span> i64</span>)</span> -></span> i64</span> {</span></span> let</span> y</span> =</span> if</span> m</span> <=</span> 2</span> {</span> y</span> -</span> 1</span> }</span> else</span> {</span> y</span> }</span>;</span></span> let</span> era</span> =</span> if</span> y</span> >=</span> 0</span> {</span> y</span> }</span> else</span> {</span> y</span> -</span> 399</span> }</span> /</span> 400</span>;</span></span> let</span> yoe</span> =</span> y</span> -</span> era</span> </span> 400</span>;</span></span> let</span> doy</span> =</span> (</span>153</span> </span> (</span>m</span> +</span> if</span> m</span> ></span> 2</span> {</span> -</span>3</span> }</span> else</span> {</span> 9</span> }</span>)</span> +</span> 2</span>)</span> /</span> 5</span> +</span> d</span> -</span> 1</span>;</span></span> let</span> doe</span> =</span> yoe</span> </span> 365</span> +</span> yoe</span> /</span> 4</span> -</span> yoe</span> /</span> 100</span> +</span> doy</span>;</span></span> era</span> </span> 146_097</span> +</span> doe</span> -</span> 719_468</span></span> }</span></span></code></pre> Result:</strong> parse time dropped from 5.52 s to 1.50 s — 3.7× faster</strong>.</p> Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read:</p> //</span> before: fresh allocation per line</span></span> fn</span> read_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>)</span> -></span> Result</span><</span>Option</span><</span>String</span>></span>></span> {</span></span> let</span> mut</span> line</span> =</span> String</span>::</span>new</span>(</span>)</span>;</span> //</span> ← new every time</span></span> reader</span>.</span>read_line</span>(</span>&</span>mut</span> line</span>)</span>?</span>;</span></span> ...</span></span> }</span></span></code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer:</p> //</span> after: capacity is retained</span></span> while</span> let</span> Some</span>(</span>Entry</span> {</span> mut</span> line</span>,</span> mut</span> reader</span>,</span> file_idx</span>,</span> ..</span> }</span>)</span> =</span> heap</span>.</span>pop</span>(</span>)</span> {</span></span> out</span>.</span>write_all</span>(</span>line</span>.</span>as_bytes</span>(</span>)</span>)</span>?</span>;</span></span> line</span>.</span>clear</span>(</span>)</span>;</span> //</span> ← keeps the heap allocation</span></span> if</span> fill_line</span>(</span>&</span>mut</span> *</span>reader</span>,</span> &</span>mut</span> line</span>)</span>?</span> {</span></span> let</span> ts</span> =</span> parse_clf_timestamp</span>(</span>&</span>line</span>)</span>.</span>unwrap_or_else</span>(</span>sentinel</span>)</span>;</span></span> heap</span>.</span>push</span>(</span>Entry</span> {</span> ts</span>,</span> file_idx</span>,</span> line</span>,</span> reader</span> }</span>)</span>;</span></span> }</span></span> }</span></span></code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path.</p> Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust:</p> #</span> Cargo.toml</span></span> mimalloc</span> =</span> {</span> version</span> =</span> "</span>0.1</span>"</span>,</span> default-features</span> =</span> false</span> }</span></span></code></pre>//</span> main.rs</span></span> use</span> mimalloc</span>::</span>MiMalloc</span>;</span></span> #</span>[</span>global_allocator</span>]</span></span> static</span> GLOBAL</span>:</span> MiMalloc</span> =</span> MiMalloc</span>;</span></span></code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well.</p> Result:</strong> 5.55 s → 5.39 s — a further ~3% improvement.</p> Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else.</p> Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together.</p> 4 MiB read buffers</strong> — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time.</p> SIMD newline search via memchr</code></strong> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>:</p> fn</span> fill_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>,</span> buf</span>:</span> &</span>mut</span> String</span>)</span> -></span> Result</span><</span>bool</span>></span> {</span></span> loop</span> {</span></span> let</span> available</span> =</span> reader</span>.</span>fill_buf</span>(</span>)</span>?</span>;</span></span> if</span> available</span>.</span>is_empty</span>(</span>)</span> {</span> return</span> Ok</span>(</span>false</span>)</span>;</span> }</span></span> match</span> memchr</span>(</span>b</span>'</span>\</span>n</span>'</span>,</span> available</span>)</span> {</span></span> Some</span>(</span>pos</span>)</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>&</span>available</span>[</span>..</span>pos</span> +</span> 1</span>]</span>)</span>?</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>pos</span> +</span> 1</span>)</span>;</span></span> return</span> Ok</span>(</span>true</span>)</span>;</span></span> }</span></span> None</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>available</span>)</span>?</span>)</span>;</span></span> let</span> len</span> =</span> available</span>.</span>len</span>(</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>len</span>)</span>;</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Result:</strong> 5.39 s → 4.43 s — another 18% improvement</strong>, bringing the total to 2.26× faster</strong> than the C original.</p> A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel:</p> Thread 1: read + decompress + parse → channel ─┐</span></span> Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout</span></span> ... ─┘</span></span></code></pre> For compressed files</strong> (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel.</p> For plain-text files</strong> on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower</em> than the sequential version with large buffers.</p> Takeaway:</strong> threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first.</p> Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines:</p> Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</strong></td> 4.43 s</strong></td> 3.61 s</strong></td> 0.81 s</strong></td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</span></span></code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload:</p> </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</strong></td> 1.6 MiB</strong></td> 30.7 MiB</strong></td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory</strong> — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained.</p> System time</strong> — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically.</p> CPU</strong> — Both tools are single-threaded and max out one core at 99%.</p> Takeaways🔗</a></h2> Five lessons from this project:</p> 1. Measure before optimizing.</strong> Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined.</p> 2. Libraries are great until they aren’t.</strong> jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×.</p> 3. Sometimes the allocator is the last few percent.</strong> After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying.</p> 4. Buffer size matters more than you think.</strong> Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better.</p> 5. Parallelism requires the right bottleneck.</strong> Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower</em>. Threading is a tool, not a universal solution.</p> 6. Ownership is not a burden.</strong> Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free.</p> The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this.</p>

Alek's Blog - parsing

mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster