Which Tool Mirrors a Cosign-Signed Image into a Private Registry?

2026-07-06T00:00:00+00:00

regctl</code> is the most complete option — it exposes the legacy tag-based signature scheme and the modern OCI 1.1 referrers API as two separate, explicit flags. oras cp -r</code> is the second choice, and the tool cosign’s own project points to now that cosign copy</code> is deprecated. oc-mirror v2</code> looks like it should be on this list but actually solves a different, unrelated signature problem.

The problem🔗</a></h2> Verifying a signature only works if the signature actually travels with the image. Anyone mirroring a Cosign-signed image into a private or air-gapped registry needs the copy step itself to carry the signature along, not just the layers — and that is not automatic. Some tools drop it silently, some need a config flag turned on first, and one widely-used tool solves a related but entirely different problem. k8s-scale-app-rs</a> is the worked example throughout this post — its own container images are signed keylessly with cosign, carry an SBOM and SLSA build provenance, and its “Verifying a published image”</a> section shows what verification actually checks. This post is about what happens before verification: getting the signature into the registry the verifier will actually query. A short, sourced comparison, checked directly against each project’s docs/source (as of July 2026): A naming trap worth knowing first🔗</a></h2> Red Hat’s oc-mirror v2</code> — the tool for mirroring into disconnected OpenShift installs — has its own --secure-policy</code>/--remove-signatures</code>/--registries.d</code> flags, but they implement Red Hat’s older “simple signing” scheme: a policy.json</code> plus GPG keys plus an external sigstore:</code> URL (e.g. mirror.openshift.com/pub/openshift-v4/signatures</code> for OpenShift release images). That predates, and has nothing to do with, the CNCF Sigstore project this post’s cosign workflow uses — same word, unrelated mechanism. oc-mirror v2</code> went GA in OpenShift 4.18 (release notes</a>), but its ImageSetConfiguration</code> API is still v2alpha1</code>, and none of Red Hat’s own walkthroughs for it touch Cosign-style signatures at all. Tool comparison🔗</a></h2> Tool</th> Legacy tag-based signature (sha256-<digest>.sig</code>)</th> OCI 1.1 referrers (modern Cosign/SBOM/attestations)</th> Note</th></tr></thead> regctl image copy</code></a></td> --digest-tags</code></td> --referrers</code></td> only tool here with separate, explicit flags for both schemes</td></tr> oras cp -r</code></a></td> via -r</code></td> -r</code> (marked “Preview”)</td> --from/to-distribution-spec</code> handles referrers-tag vs. referrers-API mismatches between registries</td></tr> skopeo copy</code></td> opt-in</td> not supported</td> needs use-sigstore-attachments: true</code> under the default-docker</code> key in registries.d</code>, on both source and destination</td></tr> Zot registry sync</code></td> syncLegacyCosignTags</code> (default true</code>)</td> only with preserveDigest: true</code></td> without preserveDigest</code>, Zot re-encodes Docker-format images to OCI on sync, which changes the digest and detaches the signature</td></tr> cosign copy</code></td> yes</td> no</td> deprecated as of sigstore/cosign#4681</a> (merged 2026-02-04) — the tool itself now points to oras copy -r</code> instead</td></tr> crane copy</code></td> only incidentally, via --all-tags</code></td> no</td> plain manifest/layer copier, no signature awareness at all</td></tr> Harbor replication</td> per Harbor’s own docs, yes</td> open gap (goharbor/harbor#23210</a>)</td> works for the legacy scheme; OCI 1.1 referrers replication is not yet reliable</td></tr> </tbody></table> regctl</code> is the only tool in this comparison that exposes both signature schemes as distinct, intentional flags rather than as a side effect of copying “everything.” For a registry pair that already speaks the OCI 1.1 referrers API end to end, oras cp -r</code> is the other option worth checking — it’s what the cosign project’s own deprecation notice for cosign copy</code> points to. Example: mirroring with regctl🔗</a></h2> Here is an example command to mirror k8s-scale-app-rs</code> itself, signature included, using regctl</code>: regctl image copy \ --digest-tags --referrers \ ghcr.io/git001/k8s-scale-app-rs:v1.0.0 \ my-private-registry.example.com/k8s-scale-app-rs:v1.0.0</code></pre>Sources🔗</a></h2> sigstore/cosign PR #4681 — “Deprecate cosign copy”</a></li> sigstore/cosign#4335 — Complete OCI 1.1 Referrers API Support Across All Cosign Commands</a></li> sigstore/cosign#4564 — cosign verify fails after cosign copy (JFrog Artifactory)</a></li> podman-container-tools/skopeo#2061 — sigstore signature copying and use-sigstore-attachments</code></a></li> regctl image copy</code> reference</a></li> oras cp</code> command reference</a></li> goharbor/harbor#23210 — OCI 1.1 referrers not replicated</a></li> Zot registry: mirroring/sync documentation</a></li> OpenShift Container Platform 4.18 release notes</a></li> openshift/oc-mirror — signature-verification.md</a></li> </ul> haproxy-spoe-rs: Deployment 2026-04-12T00:00:00+00:00 This post covers deploying the haproxy-spoe-rs</a> SPOA agent in production. It is a companion to the architecture post</a> which explains the library design. The agent and HAProxy run as separate services — HAProxy connects to the agent over TCP. This keeps their lifecycles independent and allows scaling each side separately. Running pdns_recursor as a root-independent validating resolver 2026-04-06T00:00:00+00:00 PowerDNS Recursor</a> is a mature, production-grade recursive DNS resolver. This post documents how to configure it to bootstrap directly from a local root.zone</code> file — so it never needs to reach the root name servers at runtime — with full DNSSEC validation producing the ad</code> flag in responses. Running a validating DNS recursor from the root zone with Hickory DNS 2026-04-04T00:00:00+00:00 Hickory DNS</a> is a Rust-based DNS implementation that covers the full stack: resolver, recursor, authoritative server, and DNSSEC. This post documents how to run it as a full recursive resolver — starting directly from the root zone, with DNSSEC validation and encrypted upstream transport — based on the changes developed in the recurser-from-root-zone</code></a> branch. vigil-log-relay: Kubernetes Log Collection Without a DaemonSet 2026-03-22T00:00:00+00:00 Every team running workloads on Kubernetes eventually faces the same question: how do you get logs out of your pods and into your log aggregation stack? The standard answer is a DaemonSet. It works, but it comes with a cost. vigil-log-relay</a> is a different answer. vigil-rs: A Rust Service Supervisor for Containers 2026-03-21T00:00:00+00:00 Every container that runs more than one process needs a supervisor. Most reach for dumb-init</code>, tini</code>, s6-overlay</code>, or supervisord</code>. None of them felt quite right for what we needed, so we wrote vigil-rs. This post walks through what github.com/git001/vigil-rs</a> is, why we built it, and how it works internally. mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster 2026-03-15T00:00:00+00:00 mergelog</a> is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. The original version 4.5 was written in C around 2000–2001, weighs in at ~450 lines, and does exactly what it promises — reliably and fast. This post documents the journey from mergelog-4.5 to mergelog-rs</a>: what changed, what was surprising, and how six targeted optimizations made the Rust binary 2.26× faster than the C original. Full respect and honour to Bertrand Demiddelaer, who created mergelog back in 2000. Writing a tool this fast, this correct, and this compact in C — with no external dependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his shoulders. The Original: mergelog 4.5🔗</a></h2> The C program reads any number of log files in Common Log Format (CLF) / NCSA Combined Format: %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"</code></pre> Example line: 93.184.216.34 - - [15/Mar/2026:12:00:00 +0100] "GET /index.html HTTP/1.1" 200 1234 "-" "curl/7.88"</code></pre> The algorithm in 4.5 is straightforward: find the earliest timestamp across all files, then iterate second-by-second from there to the end, writing all matching lines at each step. That is O(timespan × K) — slow when logs span multiple years or many files are involved. Gzip support exists, but as a separate binary (zmergelog</code>) compiled with -DUSE_ZLIB</code>. The Rust Rewrite: What Changed🔗</a></h2> 1. Algorithm: K-Way Merge with a Binary Heap🔗</a></h3> Instead of time-based iteration, mergelog-rs uses a k-way merge with a min-heap (std::collections::BinaryHeap</code> with reversed ordering): Seed the heap with the first line from each file</li> Main loop: pop the smallest timestamp, write the line, push the next line from the same file</li> Complexity: O(N log K) — independent of the time span of the logs</li> </ul> while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre>2. Magic-Byte Detection Instead of File Extension🔗</a></h3> The original detects gzip by binary type (zmergelog</code> vs. mergelog</code>). mergelog-rs reads the first 6 bytes of each file and detects the format automatically: Format</th> Magic Bytes</th></tr></thead> gzip</td> 1F 8B</code></td></tr> bzip2</td> 42 5A 68</code> (BZh</code>)</td></tr> xz</td> FD 37 7A 58 5A 00</code></td></tr> zstd</td> 28 B5 2F FD</code></td></tr> </tbody></table> pub fn detect_from_bytes(magic: &[u8]) -> Compression { if magic.len() >= 2 && magic[..2] == [0x1f, 0x8b] { Compression::Gz } else if magic.len() >= 3 && magic[..3] == [0x42, 0x5a, 0x68] { Compression::Bz2 } else if magic.len() >= 6 && magic[..6] == [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00] { Compression::Xz } else if magic.len() >= 4 && magic[..4] == [0x28, 0xb5, 0x2f, 0xfd] { Compression::Zstd } else { Compression::None } }</code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed. 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output. mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone. Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>: ──── profile_timing (41.65M lines) ────────────────── I/O (read_line) 3.91 s 32.4% Parse (jiff strptime) 5.52 s 45.7% ← bottleneck Heap (push/pop) 1.97 s 16.3% Write (write_all) 0.69 s 5.7% ───────────────────────────────────────────────── Σ 12.08 s</code></pre> Nearly half the time was in the parser. The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast. This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser. Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed: [DD/Mon/YYYY:HH:MM:SS ±HHMM] 0123456789012345678901234 5</code></pre> A hand-written parser only needs to: Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication: fn days_since_epoch(y: i64, m: i64, d: i64) -> i64 { let y = if m <= 2 { y - 1 } else { y }; let era = if y >= 0 { y } else { y - 399 } / 400; let yoe = y - era * 400; let doy = (153 * (m + if m > 2 { -3 } else { 9 }) + 2) / 5 + d - 1; let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; era * 146_097 + doe - 719_468 }</code></pre> Result: parse time dropped from 5.52 s to 1.50 s — 3.7× faster. Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read: // before: fresh allocation per line fn read_line(reader: &mut dyn BufRead) -> Result<Option<String>> { let mut line = String::new(); // ← new every time reader.read_line(&mut line)?; ... }</code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer: // after: capacity is retained while let Some(Entry { mut line, mut reader, file_idx, .. }) = heap.pop() { out.write_all(line.as_bytes())?; line.clear(); // ← keeps the heap allocation if fill_line(&mut *reader, &mut line)? { let ts = parse_clf_timestamp(&line).unwrap_or_else(sentinel); heap.push(Entry { ts, file_idx, line, reader }); } }</code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path. Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust: # Cargo.toml mimalloc = { version = "0.1", default-features = false }</code></pre>// main.rs use mimalloc::MiMalloc; #[global_allocator] static GLOBAL: MiMalloc = MiMalloc;</code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well. Result: 5.55 s → 5.39 s — a further ~3% improvement. Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else. Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together. 4 MiB read buffers — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time. SIMD newline search via memchr</code> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>: fn fill_line(reader: &mut dyn BufRead, buf: &mut String) -> Result<bool> { loop { let available = reader.fill_buf()?; if available.is_empty() { return Ok(false); } match memchr(b'\n', available) { Some(pos) => { buf.push_str(std::str::from_utf8(&available[..pos + 1])?); reader.consume(pos + 1); return Ok(true); } None => { buf.push_str(std::str::from_utf8(available)?); let len = available.len(); reader.consume(len); } } } }</code></pre> Result: 5.39 s → 4.43 s — another 18% improvement, bringing the total to 2.26× faster than the C original. A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel: Thread 1: read + decompress + parse → channel ─┐ Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout ... ─┘</code></pre> For compressed files (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel. For plain-text files on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower than the sequential version with large buffers. Takeaway: threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first. Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines: Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</td> 4.43 s</td> 3.61 s</td> 0.81 s</td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload: </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</td> 1.6 MiB</td> 30.7 MiB</td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained. System time — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically. CPU — Both tools are single-threaded and max out one core at 99%. Takeaways🔗</a></h2> Five lessons from this project: 1. Measure before optimizing. Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined. 2. Libraries are great until they aren’t. jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×. 3. Sometimes the allocator is the last few percent. After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying. 4. Buffer size matters more than you think. Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better. 5. Parallelism requires the right bottleneck. Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower. Threading is a tool, not a universal solution. 6. Ownership is not a burden. Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free. The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this. AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser 2026-03-12T00:00:00+00:00 AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser🔗</a></h1> Background🔗</a></h2> AWFFull 4.0.0</a> is a web server log analyser. It reads Apache/nginx/Squid access logs and produces HTML reports with graphs showing traffic, top URLs, referrers, user agents, country statistics, and more. It was originally forked from The Webalizer</a> by Bradford L. Barrett (1997–2001), extended as AWFFull by Steve McInerney from 2004, and actively developed until 2008, when it was last released as version 3.10.2. It still works well in 2026 — the core log-parsing logic is solid and the output is clean. But the codebase had accumulated 18 years of code: EOL libraries, dead links, obsolete command-line flags, a broken GeoIP implementation, and a very old config.guess</code>. Time for a proper cleanup. This post covers what changed in AWFFull 4.0.0 and why. Is a Static Web Log Analyser Still Useful?🔗</a></h2> Good question. The short answer is: yes — and arguably more so than before. Modern web analytics platforms — Google Analytics, Matomo, Plausible, Grafana with Loki, ELK stacks — are excellent for live dashboards and drill-down exploration. But they share a common weakness: they are slow when you need to process large volumes of historical log data and render the results as a complete set of graphs and tables. Feed a year of Apache logs (say, 500 million lines across 50 GB of gzipped files) into an ELK pipeline and you’ll be waiting a long time for the data to be ingested, indexed, and ready to query. Then wait again each time you open a dashboard. Grafana rendering a year’s worth of nginx traffic across 30 panels? Expect seconds of load time per page, per query. AWFFull’s approach is different: it processes the raw log files directly, keeps all state in a compact binary history file, and writes static HTML + PNG output that loads instantly in any browser — even on a machine from 2010, over a slow connection, or offline entirely. There is no query layer, no index, no database, no running server. The report is just files. For the typical use case — a sysadmin or developer who wants a monthly overview of who accessed their server, from where, with what agents, hitting which URLs — this is exactly the right trade-off: Fast to generate: processing a month of logs takes seconds, not minutes</li> Instant to view: static HTML, no round-trips, no query engine</li> Self-contained: the report directory can be archived, moved, or served from any static file server</li> Privacy-friendly: no third-party JavaScript, no tracking pixels, no data leaving your infrastructure — the raw IP addresses stay in your own logs</li> Works on air-gapped systems: nothing requires internet access at runtime</li> </ul> It also complements rather than replaces live monitoring. You would typically use Prometheus/Grafana (or similar) to watch your infrastructure in real time, and AWFFull to produce the monthly historical report that goes into a ticket or a management summary. Shipping Access Logs to Object Storage and Running in Containers🔗</a></h3> The operational details — how to ship access logs to S3 or Azure Blob with Fluent Bit, Vector, or Filebeat/Logstash, how to handle concurrent writes and log loss at high traffic, and how to run AWFFull as a stateless container against object storage — are covered in the companion post: AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers</a> The Big Ticket Items🔗</a></h2> PCRE → PCRE2🔗</a></h3> AWFFull used libpcre (PCRE version 1) for log-line regex parsing. PCRE 1 has been officially end-of-life since 2012 and is being dropped by distributions. We migrated to PCRE2, the current maintained successor with a cleaner API and better Unicode support. The change touched parser.c</code>, common.h</code>, and configure.ac</code>. GeoIP: From Legacy .dat</code> to libmaxminddb🔗</a></h3> The old code used libGeoIP with a proprietary binary .dat</code> format that MaxMind deprecated in 2019. Country lookups were broken for anyone who hadn’t kept a seven-year-old database around. AWFFull 4.0.0 replaces this with libmaxminddb, the open standard MMDB format used by both MaxMind (GeoLite2) and DB-IP. The migration is transparent — IPv4 and IPv6 are both handled by a single MMDB_lookup_string()</code> call, with no separate code paths needed. Free databases updated monthly: Country: DB-IP Country Lite</a></li> ASN: DB-IP ASN Lite</a></li> </ul> ASN Statistics: New Table and Pie Chart🔗</a></h3> Once you have a MMDB-capable lookup layer, adding ASN (Autonomous System Number) statistics is almost free. A second database (GeoASNDatabase</code>) maps IPs to their network operator — think “Cloudflare”, “Amazon AWS”, “Deutsche Telekom”. The report now shows a Top ASNs table and pie chart, answering the question “which networks send the most traffic to my site?” GeoASN yes GeoASNDatabase /usr/share/GeoIP/dbip-asn-lite.mmdb</code></pre> Performance note: both GeoIP and GeoASN lookups happen in the output phase, once per unique IP address — not once per log line. On a typical site the overhead is negligible. Hyperscan: SIMD Pattern Matching🔗</a></h3> AWFFull supports extensive pattern lists for ignoring, grouping, and hiding hosts, URLs, referrers, and agents. The original code used a Boyer-Moore-Horspool linear scan: each log record was matched against every pattern in turn. When Intel Hyperscan (libhyperscan</code>) is installed, AWFFull now compiles all patterns from a given list into a single Hyperscan database and matches them in one SIMD pass per record. For large pattern lists — say, 50+ bot/spider agent patterns — this is significantly faster. Hyperscan is auto-detected at configure time and falls back to the existing BMH path if unavailable. sudo apt install libhyperscan-dev ./configure && make</code></pre>mimalloc: Better Heap Performance🔗</a></h3> For large log files with millions of unique hosts, URLs, and referrers, heap allocation overhead becomes measurable. Adding mimalloc (Microsoft’s high-performance allocator) as an optional drop-in was straightforward via the explicit mi_calloc</code> API. Auto-detected at configure time. XXH3 Hash Function🔗</a></h3> The internal hash table used djb2, a good general-purpose hash from 1991. AWFFull 4.0.0 replaces it with XXH3 (via the bundled xxhash.h</code>), which offers better distribution and significantly higher throughput on modern CPUs. The table load-factor was also tuned to reduce collisions. User-Visible Improvements🔗</a></h2> Date Stamp on Graphs🔗</a></h3> Every generated PNG graph now shows the current date (YYYY-MM-DD</code>) in the top-right corner, aligned with the title baseline. Small but useful when you’re looking at archived reports six months later. Next / Previous Month Navigation🔗</a></h3> Monthly report pages now show navigation links at the top: ← February 2026 | [Index] | April 2026 →</code></pre> Links are derived from the history file — only months that actually exist in the database are linked. Updated Browser/Bot Detection🔗</a></h3> The sample.conf</code> browser/agent detection list dated from 2005 and contained entries like “Netscape 4.5”, “Internet Explorer 3.0 (Win95)”, and “MSNBot” — none of which have been seen in the wild for a decade or more. The list has been completely rewritten: Removed: Netscape 1–4.x, Internet Explorer 3–7, Firebird, Galeon, Camino, msnbot (all entries) Added: Chrome/Chromium, Edge (Chromium), Brave, Samsung Browser, Firefox, Safari, Opera (OPR/), modern search bots (Bingbot, DuckDuckBot, YandexBot, Baidu, Applebot, AhrefsBot, SemrushBot), AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Amazonbot), and CLI tools (curl, Wget, python-requests). Build System and Tooling🔗</a></h2> CLI Cleanup: 14 Flags Removed🔗</a></h3> AWFFull inherited many single-letter flags from the original Webalizer. Flags like -G</code> (suppress hourly graph), -L</code> (suppress legend), -M</code> (mangle agents), -x</code> (HTML extension), and -I</code> (index alias) had exact equivalents in the config file. At version 4.0.0 — a major version bump — these were removed from the CLI, along with three fully deprecated no-op flags (-d</code>, -q</code>, -Q</code>). The --help</code> output now documents the new verbosity levels (-v</code> through -vvvvv</code>) and includes a note pointing config-file-only settings to awffull.conf(5)</code>. Man Pages: DocBook XML → Pandoc Markdown🔗</a></h3> The existing man pages were written in DocBook XML 4.5 and built with xsltproc</code> — a tool that requires xsltproc</code>, docbook-xsl</code>, and several XML schema packages, none of which are installed by default on most systems in 2026. The man pages are now maintained as pandoc Markdown (doc/awffull.1.md</code>, doc/awffull.conf.5.md</code>) and built with: pandoc -s -t man awffull.1.md -o awffull.1</code></pre> configure.ac</code> auto-detects pandoc. Pre-built man pages remain in the repository as a fallback for systems without pandoc. The DocBook XML sources are retained in doc/</code> for historical reference. Translations: Quality Over Quantity🔗</a></h3> AWFFull shipped with 33 language files, most of which were approximately 29% complete and had not been updated since 2008. Rather than shipping incomplete translations that silently fall back to English mid-report, we reduced to 6 fully complete languages (German, Finnish, Italian, Swedish, Brazilian Portuguese, Indonesian) and brought all of them to 100% (524/524 strings). The .pot</code> template was regenerated with xgettext</code> to include all new strings added in 4.0.0 (GeoASN messages, MaxMind DB errors, pattern matching warnings). Repository Modernisation🔗</a></h2> The project moved from a collection of legacy files to a cleaner structure: Before</th> After</th></tr></thead> README</code> + README.FIRST</code> + README.webalizer</code></td> README.md</code> (single entry point)</td></tr> ChangeLog</code> (GNU format, frozen)</td> CHANGELOG.md</code> (Keep a Changelog)</td></tr> COPYING</code></td> LICENSE</code></td></tr> TODO</code> (plain text, 2008)</td> TODO.md</code> (Markdown with [x]</code> checkboxes)</td></tr> country-codes.txt</code> (2005, informational)</td> removed</td></tr> DNS.README</code> (dead link)</td> removed</td></tr> contrib/</code> (Webalizer migration scripts)</td> removed</td></tr> config.guess</code> from 2008</td> updated to automake 1.16 (2022)</td></tr> </tbody></table> tip </div> tip </div> The full list of technical changes is available in CHANGES.md</a>. </div> </div> CI/CD Pipeline🔗</a></h2> A full GitLab CI pipeline was added covering: Build verification on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 8/9/10</li> Source and binary tarballs on every push to main</code></li> .deb</code> package (Ubuntu 24.04 amd64) on release tags</li> .rpm</code> package (AlmaLinux 9 amd64) on release tags</li> Container image pushed to GitLab Container Registry (Ubuntu 24.04 base, all optional libs included)</li> GitLab Release creation with all artifacts uploaded to the Generic Package Registry</li> </ul> Releases are triggered by pushing a semver tag: git tag v4.0.0 git push --tags</code></pre> The container image is available as: docker pull registry.gitlab.com/aleks001/awffull:v4.0.0 docker pull registry.gitlab.com/aleks001/awffull:latest</code></pre> The other artifacts are available on the Release Page</a> What’s Next🔗</a></h2> The TODO.md</code> tracks longer-term ideas: SVG graphs (replacing the aging libgd dependency), weekly summaries, a templating system for the HTML output, and better browser/OS reporting. Contributions welcome. The project is at https://gitlab.com/aleks001/awfull</a>. AWFFull is free software licensed under GPL v3 or later. Based on The Webalizer by Bradford L. Barrett (1997–2001). Extended by Steve McInerney (2004–2008) and Aleksandar Lazic (2026–present). AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers 2026-03-12T00:00:00+00:00 This is a companion post to AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs. Shipping Access Logs to Object Storage🔗</a></h2> Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches. danger </div> Critical requirement for AWFFull compatibility </div> The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON. </div> </div> Fluent Bit → S3🔗</a></h3> Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins. # /etc/fluent-bit/fluent-bit.conf [INPUT] Name tail Path /var/log/nginx/access.log Tag nginx.access Read_from_Head false Refresh_Interval 5 [OUTPUT] Name s3 Match nginx.access bucket my-logs region eu-central-1 # Date-partitioned path — one directory per day. # $UUID is a Fluent Bit built-in variable; it ensures unique filenames # when multiple instances run concurrently (e.g. multiple Pods). s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz compression gzip total_file_size 50M # flush when file reaches 50 MB upload_timeout 10m # or after 10 minutes, whichever comes first use_put_object On</code></pre> For Azure Blob replace the output section: [OUTPUT] Name azure_blob Match nginx.access account_name mystorageaccount shared_key ${AZURE_STORAGE_KEY} container_name nginx-logs # The azure_blob plugin does not support $UUID in the path parameter. # Uniqueness is guaranteed because Fluent Bit appends an internal # timestamp + sequence counter to each blob name automatically. path nginx/%Y/%m/%d auto_create_container On tls On compress gzip</code></pre> Vector → S3 / Azure Blob🔗</a></h3> Vector</a> offers more flexibility for transformations and supports both sinks natively: # vector.yaml sources: nginx_access: type: file include: - /var/log/nginx/access.log read_from: end # only new lines; use 'beginning' for backfill sinks: s3_logs: type: aws_s3 inputs: [nginx_access] bucket: my-logs region: eu-central-1 # Vector automatically appends a UUID to key_prefix — no explicit $UUID needed. # Multiple instances writing concurrently will never collide. key_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" compression: gzip encoding: codec: text # raw lines — no JSON wrapping batch: max_bytes: 52428800 # 50 MB timeout_secs: 600 # or 10 minutes</code></pre> For Azure Blob, replace the sink: azure_blob_logs: type: azure_blob inputs: [nginx_access] connection_string: "${AZURE_STORAGE_CONNECTION_STRING}" container_name: nginx-logs # Same as S3: Vector appends a UUID automatically — uniqueness is built-in. blob_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" encoding: codec: text compression: gzip</code></pre> Filebeat → Logstash → S3🔗</a></h3> Filebeat</a> is the standard shipper in the Elastic Stack. It has an nginx module</a> that parses, enriches and structures the access log — but do not use the nginx module for AWFFull. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format. Instead, use the generic filestream</code> input with no parsing: # /etc/filebeat/filebeat.yml filebeat.inputs: - type: filestream id: nginx-access paths: - /var/log/nginx/access.log # No parsers, no module — raw lines only parsers: [] output.logstash: hosts: ["logstash:5044"]</code></pre> Filebeat has no native S3 output. Use Logstash as the intermediary with the logstash-output-s3</code> plugin: # /etc/logstash/conf.d/nginx-s3.conf input { beats { port => 5044 } } filter { # No grok, no parsing — pass the raw message through unchanged } output { s3 { bucket => "my-logs" region => "eu-central-1" prefix => "nginx/%{+YYYY}/%{+MM}/%{+dd}/" codec => line # raw lines, one per file encoding => "gzip" size_file => 52428800 # 50 MB per file time_file => 10 # or rotate after 10 minutes # Credentials from IAM role / environment } }</code></pre> The codec => line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default json_lines</code> codec would break compatibility. Note: Filebeat also supports shipping to Azure via Logstash with the logstash-output-azure_blob_storage</a> community plugin, using the same pattern. Concurrent Write Safety🔗</a></h3> When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: can this corrupt the data? Short answer: no — but data loss is possible if you don’t use unique keys. S3 and Azure Blob both guarantee atomic PUT semantics at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible. The real risk is silent overwrite: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost. Here is how the tools described above handle this: Tool</th> Mechanism</th> Multiple instances</th></tr></thead> Fluent Bit → S3</td> $UUID</code> in s3_key_format</code></td> Safe — each instance generates a unique key</td></tr> Vector → S3</td> UUID appended automatically to key_prefix</code></td> Safe — built-in</td></tr> Vector → Azure Blob</td> UUID appended automatically to blob_prefix</code></td> Safe — built-in</td></tr> Logstash → S3</td> hostname + timestamp</code> in the generated filename</td> Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> Fluent Bit → Azure Blob (append blob)</td> Append blob type with atomic server-side appends</td> Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> Fluent Bit → Azure Blob (block blob)</td> Path parameter sets the blob name</td> Unsafe if multiple instances use the same path</code> without a unique suffix</td></tr> </tbody></table> Practical rule: always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key. Log Loss at High-Traffic Sites🔗</a></h3> Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are: 1. Memory buffer overflow Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either drops new lines silently or applies backpressure upstream — which can slow down the web server itself. 2. Network outage or S3/Azure throttling Object storage APIs return 503 Slow Down</code> (S3) or 429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload. 3. Shipper crash without offset tracking If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage). 4. Log rotation race logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost. Each tool has a different answer to these problems: Tool</th> Persistence</th> At-least-once guarantee</th></tr></thead> Fluent Bit</td> Filesystem buffer (storage.type filesystem</code>)</td> Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> Vector</td> Disk buffer (type: disk</code> in buffer config)</td> Yes — survives restarts; default is in-memory only</td></tr> Logstash</td> Persistent queue (queue.type: persisted</code>)</td> Yes — survives restarts; default is in-memory only</td></tr> Filebeat</td> Registry file tracks read offset per file</td> Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated and deleted before Filebeat reads to the end, the unread bytes are lost. Recommendations for high-traffic sites: Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> Set a generous retry policy and back-off interval for S3/Azure upload errors</li> Configure log rotation with a delaycompress</code> / rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool. Fluent Bit — Filesystem Buffer + Retry🔗</a></h4> # /etc/fluent-bit/fluent-bit.conf [SERVICE] Flush 5 Daemon Off Log_Level info # Filesystem buffering — chunks are written to disk before upload. # Data survives Fluent Bit restarts and network outages up to the limit. storage.path /var/lib/fluent-bit/buf/ storage.sync normal # fsync after each chunk write storage.checksum off storage.max_chunks_up 128 # max chunks in memory at once [INPUT] Name tail Path /var/log/nginx/access.log Tag nginx.access Read_from_Head false Refresh_Interval 5 # Offset database — Fluent Bit remembers the last read position. # Without this, every restart re-reads the file from the beginning. DB /var/lib/fluent-bit/nginx-access.db # Use filesystem storage for this input's chunks (not memory) storage.type filesystem [OUTPUT] Name s3 Match nginx.access bucket my-logs region eu-central-1 s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz compression gzip total_file_size 50M upload_timeout 10m use_put_object On # Retry: up to 5 attempts with exponential back-off (max ~160 s between tries). # Set False for unlimited retries (use with caution — can block the pipeline). Retry_Limit 5 # Cap the on-disk buffer for this output at 1 GB. # When full, oldest chunks are dropped to make room. storage.total_limit_size 1G</code></pre> The DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if logrotate</code> renames access.log</code> to access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file. Vector — Disk Buffer + Retry🔗</a></h4> # /etc/vector/vector.yaml sources: nginx_access: type: file include: - /var/log/nginx/access.log read_from: end # Vector also writes a checkpoint file (similar to Filebeat's registry) # to /var/lib/vector/ by default — survives restarts automatically. sinks: s3_logs: type: aws_s3 inputs: [nginx_access] bucket: my-logs region: eu-central-1 key_prefix: "nginx/{{ now() | strftime('%Y/%m/%d') }}/" compression: gzip encoding: codec: text batch: max_bytes: 52428800 # 50 MB per object timeout_secs: 600 # or 10 minutes, whichever comes first # Persistent disk buffer — events are written to disk before upload. # Survives Vector restarts; default is type: memory (lost on crash). buffer: type: disk max_size: 1073741824 # 1 GB when_full: block # apply backpressure instead of dropping events # Retry policy for S3 upload failures (503, 429, network errors). request: retry_attempts: 10 # max upload attempts per batch retry_initial_backoff_secs: 1 # first retry after 1 s retry_max_duration_secs: 300 # cap back-off at 5 minutes timeout_secs: 60 # per-request timeout rate_limit_num: 100 # max concurrent requests</code></pre> when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog. Filebeat + Logstash — Registry + Persistent Queue🔗</a></h4> Filebeat (/etc/filebeat/filebeat.yml</code>): filebeat.inputs: - type: filestream id: nginx-access paths: - /var/log/nginx/access.log parsers: [] # no parsing — raw lines only # Filebeat writes a registry to /var/lib/filebeat/registry/ that records # the inode, byte offset, and identifier for every tracked file. # On restart it resumes exactly where it stopped — no duplicates, no gaps. filebeat.registry.path: /var/lib/filebeat/registry # Internal memory queue — tune for throughput vs. latency. # These events are lost on crash; durability comes from the registry, # not from this queue. queue.mem: events: 4096 flush.min_events: 512 flush.timeout: 5s output.logstash: hosts: ["logstash:5044"] # Retry connecting to Logstash — Filebeat will keep trying indefinitely. # Events accumulate in the queue while Logstash is unreachable. timeout: 30 max_retries: 3 # per batch; 0 = retry forever bulk_max_size: 2048</code></pre> Logstash (/etc/logstash/logstash.yml</code>): # Persistent queue — events are written to disk between the input and filter/ # output stages. Logstash survives restarts without losing in-flight events. queue.type: persisted queue.max_bytes: 1gb queue.path: /var/lib/logstash/queue # Checkpoint frequency — lower values = more durable, more I/O. queue.checkpoint.acks: 1024 queue.checkpoint.writes: 1024 # Dead-letter queue — events that fail after all retries go here instead # of being silently dropped. Inspect with the dead_letter_queue input plugin. dead_letter_queue.enable: true dead_letter_queue.max_bytes: 1gb path.dead_letter_queue: /var/lib/logstash/dead_letter_queue</code></pre> Logstash pipeline (/etc/logstash/conf.d/nginx-s3.conf</code>): input { beats { port => 5044 } } filter { # No grok, no parsing — pass the raw message through unchanged } output { s3 { bucket => "my-logs" region => "eu-central-1" prefix => "nginx/%{+YYYY}/%{+MM}/%{+dd}/" codec => line # one raw log line per line encoding => "gzip" size_file => 52428800 # rotate object at 50 MB time_file => 10 # or after 10 minutes # Local staging directory — files are assembled here before upload. # Use a path on a real disk, not tmpfs, for crash safety. temporary_directory => "/var/lib/logstash/s3-tmp" # Retry failed uploads before giving up and sending to the DLQ. retry_count => 10 retry_delay => 1 # seconds between retries } }</code></pre> With queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped. Running AWFFull Against Object Storage Logs🔗</a></h2> info </div> Info </div> The AWFFull container downloads the log files and the previous awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed. </div> </div> Once logs are stored with date-based prefixes, fetching a specific day is a single command: From S3: # Download all log files for 12 March 2026 aws s3 sync s3://my-logs/nginx/2026/03/12/ /tmp/logs/2026-03-12/ # Process with AWFFull (pass files in chronological order) awffull -c /etc/awffull.conf $(ls /tmp/logs/2026-03-12/*.log.gz | sort)</code></pre> From Azure Blob: azcopy copy \ "https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/*" \ /tmp/logs/2026-03-12/ \ --recursive awffull -c /etc/awffull.conf $(ls /tmp/logs/2026-03-12/*.log.gz | sort)</code></pre> Downloading a full month (common for monthly reporting): aws s3 sync s3://my-logs/nginx/2026/03/ /tmp/logs/2026-03/ --include "*.gz" awffull -c /etc/awffull.conf $(find /tmp/logs/2026-03/ -name "*.log.gz" | sort)</code></pre> The sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically. Running AWFFull in a Container World🔗</a></h2> AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns: Option 1 — Persistent Volume (classical approach)🔗</a></h3> The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup. # docker-compose example services: awffull: image: registry.gitlab.com/aleks001/awfull:latest volumes: - /var/log/nginx:/logs:ro # log files (read-only) - /srv/awffull/reports:/reports # generated HTML + PNG output - /etc/awffull.conf:/etc/awffull.conf:ro command: ["-c", "/etc/awffull.conf", "/logs/access.log.gz"]</code></pre> In Kubernetes, replace the host-path mounts with a PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here. # Kubernetes CronJob skeleton apiVersion: batch/v1 kind: CronJob metadata: name: awffull-monthly spec: schedule: "0 2 1 * *" # 02:00 on the 1st of each month jobTemplate: spec: template: spec: containers: - name: awffull image: registry.gitlab.com/aleks001/awfull:latest args: ["-c", "/etc/awffull.conf", "/logs/access.log.gz"] volumeMounts: - name: logs mountPath: /logs - name: reports mountPath: /reports - name: config mountPath: /etc/awffull.conf subPath: awffull.conf volumes: - name: logs persistentVolumeClaim: { claimName: nginx-logs } - name: reports persistentVolumeClaim: { claimName: awffull-reports } - name: config configMap: { name: awffull-config } restartPolicy: OnFailure</code></pre> Trade-off: Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod). Option 2 — S3 / S3-compatible Object Storage🔗</a></h3> This option covers two related questions: Where do the generated reports go?</li> Can logs that were already shipped to S3 be fed back into AWFFull?</li> </ol> The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline. Shipping logs to S3 is standard practice. Tools like Fluent Bit</a>, Vector</a>, or even a plain logrotate</code> + aws s3 cp</code> cron job send compressed log files to a bucket with a path like s3://my-logs/nginx/2026/03/01/access.log.gz</code>. Reading them back for AWFFull is equally straightforward: #!/bin/sh # 1. Download history file (needed for incremental processing) aws s3 cp s3://my-bucket/awffull/awffull.hist /var/lib/awffull/ 2>/dev/null || true # 2. Download log files for the period to process aws s3 sync s3://my-logs/nginx/2026/03/ /tmp/logs/ --include "*.gz" # 3. Run AWFFull (reads logs, updates history, writes HTML) awffull -c /etc/awffull.conf /tmp/logs/*.gz # 4. Persist the updated history file back to S3 aws s3 cp /var/lib/awffull/awffull.hist s3://my-bucket/awffull/ # 5. Sync reports to the reports bucket / static website aws s3 sync /reports/ s3://my-reports-bucket/awffull/ --delete</code></pre> The awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well. In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all: S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist) ▲ └── IAM role / workload identity</code></pre> Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.). #!/bin/sh # entrypoint wrapper awffull -c /etc/awffull.conf /logs/access.log.gz # Sync output to S3 aws s3 sync /reports/ s3://my-bucket/awffull/ \ --delete \ --content-type "text/html" \ --exclude "*.png" \ --include "*.html" aws s3 sync /reports/ s3://my-bucket/awffull/ \ --exclude "*.html" \ --include "*.png"</code></pre> In a container, add awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables. Trade-off: Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions. Option 3 — Azure Blob Storage🔗</a></h3> Same pattern as S3, using the Azure CLI (az storage blob</code>) or azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint. Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull: #!/bin/sh STORAGE="https://<storage-account>.blob.core.windows.net" # 1. Download history file azcopy copy "${STORAGE}/awffull/awffull.hist" /var/lib/awffull/ 2>/dev/null || true # 2. Download log files for the period azcopy copy "${STORAGE}/nginx-logs/2026/03/*" /tmp/logs/ --recursive # 3. Run AWFFull awffull -c /etc/awffull.conf /tmp/logs/*.gz # 4. Persist history back to Blob azcopy copy /var/lib/awffull/awffull.hist "${STORAGE}/awffull/awffull.hist" # 5. Sync reports to static website container ($web) azcopy sync /reports/ "${STORAGE}/\$web/awffull/" \ --recursive \ --delete-destination=true</code></pre> Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image: # With workload identity (AKS) az login --identity az storage blob sync \ --source /reports/ \ --container '$web' \ --account-name <storage-account> \ --destination awffull/</code></pre> Enable the Static website feature on the Storage Account and the reports are immediately reachable at https://<storage-account>.z6.web.core.windows.net/awffull/index.html</code>. Trade-off: Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials. Choosing a Pattern🔗</a></h3> Scenario</th> Recommended approach</th></tr></thead> Simple on-prem Kubernetes</td> Persistent Volume + nginx sidecar</td></tr> AWS / EKS</td> S3 + CloudFront static website</td></tr> Azure / AKS</td> Azure Blob static website + workload identity</td></tr> Self-hosted, no cloud</td> S3 sync</td></tr> CI/CD pipeline artifact</td> Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path. AWFFull is free software licensed under GPL v3 or later. The project is at https://gitlab.com/aleks001/awfful</a>. loadgen-rs (Short): Results, Quick Start, and Tool Choice 2026-03-03T00:00:00+00:00 This is the short version. If you want profiling details, FFI internals, and full cloud deployment walkthroughs, read the full article: loadgen-rs: An HTTP Benchmark Client in Rust</a>. TL;DR🔗</a></h2> loadgen-rs</code> is a Rust benchmark client for HTTP/1.1, HTTP/2, and HTTP/3 with: one CLI for all protocols</li> JSONL/CSV output for automation</li> distributed controller/worker mode</li> scripted Deno scenarios via FFI (k6-style checks/extractors)</li> </ul> In short: very strong for reproducible automation and multi-node testing. If your only goal is absolute H3 peak throughput, h2load</code> still has an edge. Quick Start🔗</a></h2> Single-node cli🔗</a></h3> # H2 example, 10s run loadgen-rs -n 0 -D 10s -c 8 -t 4 -m 10 \ --alpn-list=h2 --insecure 'https://bench.local:8082/?s=256k'</code></pre>Single-node container🔗</a></h3> # HTTP/3 (QUIC) 10 seconds podman run -it --rm \ --network host \ --name gh-loadgen ghcr.io/git001/loadgen-rs:v0.3.0 \ -D 10s -c 8 -t 4 -m 10 \ --alpn-list=h3 \ --insecure \ 'https://bench.local:8082/?s=256k'</code></pre>Distributed🔗</a></h3> # 1) start workers bash examples/worker-start.sh # 2) run controller deno run --allow-ffi --allow-net --allow-read --allow-env \ examples/distributed.ts http://worker1:9091 http://worker2:9091</code></pre> Why distributed matters: you can scale beyond one machine and still get statistically correct merged percentiles (histogram merge, not naive p99 averaging). Benchmark Snapshot vs h2load🔗</a></h2> High-concurrency test (-c 512 -t 8 -m 8</code>, 30s): Protocol</th> h2load RPS</th> loadgen-rs RPS</th> Ratio</th></tr></thead> H1</td> 18,851</td> 19,148</td> 101.6%</td></tr> H2</td> 17,837</td> 18,025</td> 101.1%</td></tr> H3</td> 9,006</td> 6,702</td> 74.4%</td></tr> </tbody></table> Takeaway: H1/H2 throughput: effectively on par (or slightly better for loadgen-rs in this run).</li> H3 throughput: h2load remains faster.</li> Memory footprint: h2load is smaller; loadgen-rs typically used more memory in tests.</li> </ul> Which Tool Should You Use?🔗</a></h2> Need</th> Better fit</th></tr></thead> Highest H3 throughput only</td> h2load</td></tr> One tool for H1/H2/H3 + machine-readable reports</td> loadgen-rs</td></tr> Distributed multi-machine load generation</td> loadgen-rs</td></tr> Scripted scenarios with checks/extractors and correlation</td> loadgen-rs</td></tr> Lowest memory footprint</td> h2load</td></tr> </tbody></table> Next Step🔗</a></h2> For the complete deep dive (profiling, architecture, FFI API design, Terraform/Ansible workflow), continue with the full article: loadgen-rs: An HTTP Benchmark Client in Rust</a>. loadgen-rs: An HTTP Benchmark Client in Rust 2026-03-03T00:00:00+00:00 TL;DR / Executive Summary🔗</a></h2> Prefer a shorter read first? loadgen-rs (Short): Results, Quick Start, and Tool Choice</a> loadgen-rs is a Rust HTTP benchmark client that covers HTTP/1.1, HTTP/2, and HTTP/3 in one tool, outputs machine-readable JSONL/CSV, and scales from single-node runs to distributed multi-worker tests. H1/H2 throughput is competitive with h2load, while H3 still favors h2load in peak throughput.</li> loadgen-rs is strong when you want automation-friendly output and one consistent workflow across protocols.</li> Distributed mode is a first-class feature, not an afterthought: controller + workers + correct histogram merge.</li> A Deno/FFI layer enables scripted, k6-style scenarios with checks, extractors, and correlation.</li> </ul> What Is loadgen-rs?🔗</a></h2> Why I Built It🔗</a></h3> If you benchmark HTTP servers, you probably know h2load</a> from the nghttp2 project. It’s a very solid and mature tool, but there are a few pain points: HTTP/3 support is not enabled by default — h2load can do H3/QUIC, but only via a separate build variant with ngtcp2.</li> Requires a C compiler and many native libraries — this makes source builds and CI setup more cumbersome, especially across different environments.</li> </ul> I wanted a tool that tests all three HTTP protocols and outputs the results directly as JSON and easy to build. So I built loadgen-rs</a>. Core Capabilities (H1/H2/H3, JSONL, Distributed, Scripting)🔗</a></h3> Component</th> Crate</th> Why</th></tr></thead> Runtime</td> tokio (multi-thread)</td> De-facto standard for async Rust</td></tr> HTTP/1.1</td> Raw TCP + httparse</td> Maximum control, zero framework overhead</td></tr> HTTP/2</td> h2 (direct, no hyper)</td> Avoids hyper’s abstraction layer</td></tr> TLS</td> rustls (ring backend)</td> Pure Rust, no OpenSSL dependency hell</td></tr> HTTP/3</td> quinn + h3 + h3-quinn</td> QUIC stack in Rust</td></tr> CLI</td> clap (derive)</td> Ergonomic, type-safe</td></tr> Metrics</td> hdrhistogram</td> Industry standard for latency measurements</td></tr> Allocator</td> mimalloc</td> Better multi-threaded allocation performance</td></tr> </tbody></table> An early design choice was to bypass hyper</a> entirely. For H1, we use raw TCP sockets with httparse</code> for response parsing. For H2, we use the h2</code> crate directly. This eliminates one abstraction layer and gives full control over connection management, flow control tuning, and request templating. Hyper</a> is a great library, but its abstraction model didn’t fit the needs of this benchmarking tool. In addition to single-node CLI benchmarking, loadgen-rs includes distributed controller/worker execution and scripted scenario support through a native FFI layer for Deno. Runtime and Execution Model🔗</a></h3> </figure> Connection Strategy🔗</a></h4> The -c</code> and -m</code> flags follow h2load semantics: -c N</code> spawns N parallel connections distributed across worker threads</li> -m N</code> limits concurrent in-flight requests per connection (H2/H3: concurrent streams; H1: in-flight cap)</li> Total parallelism: c * m</code></li> </ul> Each worker thread runs its own current_thread</code> Tokio runtime — no cross-thread synchronization in the hot path. Workers own local HdrHistograms that are merged only at the end. Count vs. Duration Mode🔗</a></h4> Count mode (-n 1000</code>): A shared AtomicU64</code> budget that workers decrement via CAS</li> Duration mode (-D 10s</code>): A CancellationToken</code> that fires after the specified duration</li> -D</code> takes priority over -n</code> (matching h2load behavior)</li> </ul> RPS Semantics (h2load-compatible)🔗</a></h4> --rps</code> is implemented as a per-client request start rate, matching h2load</code>. Per-client target: --rps</code></li> Aggregate target: --rps * -c</code></li> </ul> This matters for comparisons: -c 10 --rps 20</code> targets roughly 200</code> started req/s overall</li> -c 10 --rps 200</code> targets roughly 2000</code> started req/s overall</li> </ul> Also note: in short count mode runs (-n</code>), achieved %</code> can appear above or below 100% because startup and shutdown overhead dominate a short elapsed window. For stable pacing comparisons, use duration mode. Output and TLS Notes🔗</a></h3> The JSONL output contains everything a Python script needs: { "proto": "h2", "url": "https://target:8443/", "clients": 8, "threads": 4, "rps": 12543.7, "latency_p50_us": 320, "latency_p90_us": 890, "latency_p99_us": 2100, "status_counts": {"200": 125437}, "status_2xx": 125437, "err_total": 0, "elapsed_s": 10.001 }</code></pre> In Python: import json results = [json.loads(line) for line in open("results.jsonl")] df = pd.DataFrame(results) df.plot(x="clients", y="rps", kind="bar")</code></pre>TLS Configuration🔗</a></h4> --insecure</code> implements a custom ServerCertVerifier</code> that accepts everything. --tls-ciphers</code> allows selecting specific cipher suites — with validation that H3 only permits TLS 1.3 ciphers. Custom CA certificates can be loaded via --tls-ca</code>. Quick Start🔗</a></h2> Single-Node Benchmark in 30 Seconds (cli)🔗</a></h3> # HTTP/1.1, 1000 requests loadgen-rs -n 1000 -c 4 -t 2 --h1 'http://bench.local:8081/?s=256k' # HTTPS loadgen-rs -n 1000 -c 4 -t 2 --h1 --insecure 'https://bench.local:8082/?s=256k' # HTTP/2, 10 seconds loadgen-rs -n 0 -D 10s -c 8 -t 4 -m 10 \ --alpn-list=h2 --insecure 'https://bench.local:8082/?s=256k' # HTTP/3 (QUIC) loadgen-rs -n 0 -D 10s -c 4 -t 2 \ --alpn-list=h3 --insecure 'https://bench.local:8082/?s=256k'</code></pre>Single-Node Benchmark in 30 Seconds (container)🔗</a></h3> If you don’t have podman</code>, you can use docker</code> instead. # HTTP/3 (QUIC) 10 seconds podman run -it --rm \ --network host \ --name gh-loadgen ghcr.io/git001/loadgen-rs:v0.3.0 \ -D 10s -c 8 -t 4 -m 10 \ --alpn-list=h3 \ --insecure \ 'https://bench.local:8082/?s=256k'</code></pre>Distributed Benchmark in 2 Minutes🔗</a></h3> # Start workers (one per machine, or locally for testing) bash examples/worker-start.sh # Run controller deno run --allow-ffi --allow-net --allow-read --allow-env \ examples/distributed.ts http://worker1:9091 http://worker2:9091</code></pre> Or programmatically: import { Controller } from "./ts/mod.ts"; const controller = new Controller({ workers: ["http://worker1:9091", "http://worker2:9091"], config: { url: "https://target:8443/", protocol: "h2", clients: 200, // split: 100 per worker threads: 4, duration_s: 30, insecure: true, }, }); const result = await controller.run(); console.log(result.merged_report.rps);</code></pre>Minimal Comparison Against h2load🔗</a></h3> # 1) H1 count mode (same request budget) h2load --h1 -n 200 -c 10 -t 8 -m 1 "https://bench.local:8082/?s=256k" loadgen-rs --h1 -n 200 -c 10 -t 8 -m 1 "https://bench.local:8082/?s=256k" # 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients * 20 rps/client) h2load --h1 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --h1 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" # 3) H2 duration mode with pacing (same effective target) h2load --alpn-list=h2 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --alpn-list h2 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" # 4) H3 duration mode with pacing (same effective target) h2load --alpn-list=h3 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --alpn-list h3 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k"</code></pre>Key Results vs h2load🔗</a></h2> After implementing and tuning the tool, I ran systematic A/B comparisons against h2load</code> on the same hardware and target server. High-Concurrency Benchmark🔗</a></h3> Setup: Duration: 30 seconds per run, 30-second cooldown between runs</li> Configuration: -c 512 -t 8 -m 8</code></li> Target: https://127.0.0.1:8082/?s=256k</code> (256 KB response body)</li> h2load runs via podman run --rm --network host localhost/h2load</code></li> </ul> Protocol</th> h2load RPS</th> loadgen-rs RPS</th> Ratio</th> h2load Latency (mean)</th> loadgen-rs Latency (mean)</th></tr></thead> H1</td> 18,851</td> 19,148</td> 101.6%</td> 27.07 ms</td> 25.69 ms</td></tr> H2</td> 17,837</td> 18,025</td> 101.1%</td> 125.59 ms</td> 223.91 ms</td></tr> H3</td> 9,006</td> 6,702</td> 74.4%</td> 450.15 ms</td> 604.17 ms</td></tr> </tbody></table> Key observations: H1: loadgen-rs is faster than h2load on both throughput and latency. The raw TCP + httparse approach with pre-built request templates pays off.</li> H2: Throughput is essentially identical. However, mean latency is ~1.8x higher in loadgen-rs due to an architectural difference (see profiling section below).</li> H3: h2load (using ngtcp2/nghttp3 in C) retains a ~25% throughput advantage over quinn/h3 in Rust.</li> </ul> To be fair, h2load was sometimes better than loadgen-rs, and vice versa. Resource Usage (CPU and Memory)🔗</a></h3> For resource fairness, both tools were run as containers and sampled with podman stats</code>: Setup: 3 repeats per point, duration=5s</code>, c=4</code>, t=2</code>, m in {1,8}</code></li> </ul> Metric</th> H1/H2</th> H3</th></tr></thead> CPU time</td> mostly +2%</code> to +8%</code></td> -1%</code> to +10%</code></td></tr> Peak memory</td> about +410%</code> to +914%</code></td> about +1393%</code></td></tr> </tbody></table> In absolute terms, peak memory was typically: h2load</code>: around 2.3 MB</code> to 3.0 MB</code></li> loadgen-rs</code>: around 15 MB</code> to 39 MB</code></li> </ul> The higher memory usage comes from Tokio’s runtime overhead, per-worker HdrHistograms, and Rust’s buffered I/O layers. In practice, even 39 MB is negligible for a benchmark client. Which Tool Should You Choose?🔗</a></h2> Decision Matrix by Use Case🔗</a></h3> Priority</th> Recommendation</th></tr></thead> H1/H2 throughput + automation</td> loadgen-rs — matches or beats h2load, with native JSONL output</td></tr> H3 peak throughput</td> h2load — ngtcp2 is ~25% faster than quinn</td></tr> Lowest memory footprint</td> h2load — 3 MB vs 15-39 MB</td></tr> Cross-protocol consistency</td> loadgen-rs — same tool, same output format for H1/H2/H3</td></tr> Multi-machine load generation</td> loadgen-rs — built-in distributed mode with correct histogram merge</td></tr> Cloud-provisioned benchmarks</td> loadgen-rs — Terraform + Ansible pipeline for Hetzner Cloud</td></tr> Distributed scripted scenarios</td> loadgen-rs — workers run multi-step checks/extractors via POST /run-script</code></td></tr> </tbody></table> Practical Recommendation🔗</a></h3> For most benchmarking workflows, the choice comes down to this: do you need the absolute highest H3 throughput, or do you want a single binary with all protocols available out of the box? Distributed Mode Overview🔗</a></h2> A single loadgen-rs process can saturate most test targets, but sometimes you need more — testing a load balancer with geographically distributed traffic, generating more requests than one machine’s NIC can handle, or simply simulating a more realistic multi-source access pattern. Architecture🔗</a></h3> The distributed mode uses a TypeScript controller (running in Deno) that orchestrates multiple worker-agents over plain HTTP/JSON — no gRPC or custom protocol needed: </figure> Each worker is a lightweight Deno HTTP server (~120 LOC) that wraps the same LoadgenFFI</code> batch API used for single-node benchmarks. The controller splits the total client count (-c</code>) evenly across workers, sets a coordinated start timestamp, and fires all jobs concurrently. The histogram merge problem🔗</a></h3> Merging latency results from distributed workers is deceptively tricky. The naive approach — averaging p99 values from each worker — is statistically wrong. If worker A has p99 = 5ms (from 50K requests) and worker B has p99 = 15ms (from 50K requests), the combined p99 is not 10ms. It depends on the full distribution shape. The correct approach is to merge the underlying histograms. Each worker serializes its HdrHistogram using V2-Deflate format (a compact binary encoding, typically 1–5 KB per histogram), base64-encodes it, and includes it in the RunReport</code> JSON: { "rps": 136949.0, "latency_p99_us": 210, "latency_hist_b64": "HISTFAAAB...", "ttfb_hist_b64": "HISTFAAAB...", "connect_hist_b64": "HISTFAAAB..." }</code></pre> The controller passes all worker reports to loadgen_merge_reports()</code> — a new FFI export that deserializes each histogram, merges them with Histogram::add()</code>, and recomputes all percentile fields from the combined distribution. This is the same merge algorithm used internally for per-thread metrics within a single process. Coordinated start🔗</a></h3> For benchmark results to be meaningful, all workers should start at the same time. The controller calculates start_at = now + 500ms</code>, includes it in each worker’s /run</code> request, and each worker sleeps until that timestamp before starting the benchmark. Clock skew under 10ms is acceptable for benchmark purposes — and on a local network or the same machine, skew is typically sub-millisecond. Minimal new code, maximum reuse🔗</a></h3> The implementation reuses the existing FFI infrastructure almost entirely: Component</th> New code</th> Reused</th></tr></thead> Rust: merge_distributed_reports()</code></td> ~120 LOC</td> Existing merge_metrics()</code> pattern</td></tr> Rust: FFI export</td> ~25 LOC</td> Existing catch_unwind</code> + error slot pattern</td></tr> TypeScript: worker-agent</td> ~120 LOC</td> LoadgenFFI</code> class</td></tr> TypeScript: Controller</td> ~150 LOC</td> mergeReports()</code> FFI wrapper</td></tr> </tbody></table> Only two new Rust dependencies were needed: base64</code> (for histogram serialization) was already an indirect dependency via reqwest. The export_histograms</code> field in BenchConfig</code> defaults to false</code>, so the existing CLI and single-node FFI path are completely unaffected. Real-world validation🔗</a></h3> Testing 2 local workers against the same target (H2, 8 total clients, 5s): Setup</th> RPS</th> p99 Latency</th></tr></thead> Single-node (-c 8 -m 4</code>)</td> 284,226</td> 185 us</td></tr> Distributed (2 × 4 clients)</td> 269,351</td> 207 us</td></tr> Ratio</td> 94.8%</td> </td></tr> </tbody></table> The ~5% overhead comes from the HTTP/JSON coordination round-trip and the coordinated start delay. In a real multi-machine deployment (where each worker has its own NIC and CPU), total throughput scales linearly — you’re no longer bound by a single machine’s resources. Scripted Scenarios Overview (Deno / k6-style)🔗</a></h2> After the initial release, I added a TypeScript/Deno layer on top of the native FFI to make scripted testing and k6-style output easier to compare. What was added🔗</a></h3> A typed Deno SDK in ts/</code>: LoadgenFFI</code> for run-level benchmarking</li> LoadgenStepFFI</code> for step/session execution</li> </ul> </li> A reusable printK6LikeSummary()</code> formatter used by the Deno examples</li> New comparison examples: examples/ab-compare.ts</code> and examples/k6/ab-compare.ts</code> (closed model)</li> examples/ab-compare-rps.ts</code> and examples/k6/ab-compare-rps.ts</code> (fixed RPS)</li> </ul> </li> A one-command comparison helper: scripts/ab_compare_report.sh</code> (prints req/s, avg, p99, fail rate, H2 check)</li> </ul> </li> </ul> Important fairness fix🔗</a></h3> One easy mistake is RPS interpretation: In loadgen-rs, --rps</code> is per client</li> Aggregate target is rps_per_client * clients</code></li> </ul> The Deno comparison script now maps a total target RPS to per-client RPS automatically (RPS / CLIENTS</code>) to match k6 constant-arrival tests. Why latency can differ even at similar req/s🔗</a></h3> Even when request rate is aligned, latency can diverge if concurrency shape is not aligned: number of active clients/VUs</li> streams per connection (-m</code> / max_streams</code>)</li> thread count and connection reuse behavior</li> </ul> In short: equal target RPS does not automatically imply equal queueing behavior. Stage profile support (k6-like)🔗</a></h3> I also added examples/ab-stages.ts</code>, which emulates stage ramps by splitting stages into short segments and running sequential FFI runs. This gives a practical equivalent for basic k6 stage workflows. Correlation path🔗</a></h3> For dynamic data and sequence scenarios, runScriptScenario()</code> supports: execution_mode: "fetch"</code> (pure Deno)</li> execution_mode: "ffi-step"</code> (native step transport)</li> </ul> The step API design and contract are documented in: docs/script-mode-ffi.md</code></li> docs/ffi-step-api.md</code></li> docs/ffi-contract.md</code></li> </ul> k6 Parity Snapshot (as of March 1, 2026)🔗</a></h3> Short answer: Deno + loadgen-rs is now close to k6 for HTTP-focused scripted workflows, but it is not yet a full 1:1 replacement. What is now very similar: Scripted scenarios with step chains, checks, and extractors</li> Correlation and dynamic data ({{var}}</code> templates, JSON/regex/header/DOM extract)</li> Cookie handling and redirect policies</li> Two execution paths: pure fetch</code> and native ffi-step</code></li> Multi-protocol support in native step mode (h1</code>, h2</code>, h3</code>)</li> k6-like terminal summary output and side-by-side comparison scripts</li> </ul> Where k6 is still stronger: More mature executor/scenario system overall (arrival/vu/stage/ramping breadth)</li> Larger built-in ecosystem (k6/http</code>, ws/browser modules, extension landscape)</li> Richer output/integration stack (cloud and common observability backends)</li> Longer production track record for large shared test suites and CI pipelines</li> </ul> Practical takeaway: If your priority is HTTP load + correlation + native H1/H2/H3 coverage with comparable CLI summaries, Deno + loadgen-rs is already highly usable. If you need the full k6 orchestration and ecosystem depth, k6 still has an edge. The distributed mode isn’t limited to raw throughput benchmarks. Workers also support POST /run-script</code> — the same scripted scenario engine used locally, but executed remotely on each worker. This means you can run multi-step request sequences with checks, extractors, cookies, and response validation across a fleet of machines. The implementation required minimal changes: the worker-agent imports runScriptScenario()</code> from the existing script_mode.ts</code> and exposes it as a new HTTP endpoint. The controller sends a ScriptScenarioConfig</code> (same type used for local script mode) to each worker and collects ScriptRunResult</code> objects with per-check pass/fail counters. A practical example ships as examples/html-check.ts</code> — a verification script that tests whether each worker can reach a target URL and parse the response correctly: const config: ScriptScenarioConfig = { vus: 3, duration_s: 5, execution_mode: "ffi-step", step_session_config: { protocol: "h2", insecure: true, response_headers: true, response_body_limit: 2_000_000, }, steps: [{ name: "html-get", method: "GET", url: targetUrl, capture_body: true, checks: { status_in: [200, 403], body_includes: ["<title>Example</title>"], header_includes: { "set-cookie": "session=" }, }, }], };</code></pre> The controller sends this config to each worker via POST /run-script</code>, then aggregates the results into a per-worker table. If the target returns both the HTML element and the cookie on every single response, the iteration count, HTML match count, and header match count should be identical — any mismatch indicates a parsing or check evaluation bug: === per-worker results === worker iterations html_match header_match ok? ------------------------------------------------------------------------------------- http://worker1:9091 62 62 62 PASS http://worker2:9091 47 47 47 PASS ------------------------------------------------------------------------------------- TOTAL 109 109 109 PASS</code></pre> All counts match perfectly across both workers. This validates the full chain: Terraform-provisioned infrastructure, Ansible-deployed workers, ffi-step</code> execution mode with reqwest on the Rust side, response header capture, and the check evaluation engine — all working end-to-end on remote machines. The header_includes</code> check type was added specifically for this use case. Unlike header_exists</code> (which only checks presence) or header_equals</code> (which requires an exact match), header_includes</code> tests whether a header value contains a substring — essential for Set-Cookie</code> headers where the full value includes expiry, path, and other attributes beyond the cookie name. Multi-Step Login/Logout Workflow🔗</a></h4> The scripted scenario engine really shines when steps depend on each other. A second distributed example — examples/roundcube-login.ts</code> — verifies a full Roundcube Webmail login/logout cycle across workers. Unlike the single-step HTML check, this scenario chains three HTTP requests with state flowing between them: steps: [ { name: "login-page", method: "GET", url: "https://mail.example.com/mail/", capture_body: true, checks: { status_in: [200], body_includes: ["Roundcube Webmail :: Welcome to Roundcube Webmail"], }, extract: [{ type: "dom", selector: 'input[name="_token"]', attribute: "value", as: "token", }], }, { name: "login-submit", method: "POST", url: "https://mail.example.com/mail/?_task=login", headers: { "content-type": "application/x-www-form-urlencoded" }, body: "_task=login&_action=login&_timezone=...&_token={{token}}&_user=...&_pass=...", capture_body: true, checks: { status_in: [200], body_includes: ["Roundcube Webmail :: Inbox", 'data-label-msg="The list is empty."'], }, extract: [{ type: "regex", pattern: '"request_token":"([^"]+)"', group: 1, as: "logout_token", }], }, { name: "logout", method: "GET", url: "https://mail.example.com/mail/?_task=logout&_token={{logout_token}}", capture_body: true, checks: { status_in: [200], body_includes: ["Roundcube Webmail :: Welcome to Roundcube Webmail"], }, }, ]</code></pre> Each step depends on extracted values from the previous one: {{token}}</code> (the CSRF token extracted from the login form’s hidden input via a DOM CSS selector — input[name="_token"]</code>) flows into the POST body, and {{logout_token}}</code> (the request_token</code> extracted from the Roundcube JavaScript via regex) flows into the logout URL. The dom</code> extractor parses the HTML response with deno-dom’s DOMParser</code> and runs querySelector</code> — far more robust than regex for structured HTML. Session cookies are managed automatically by the native cookie_jar</code> — the roundcube_sessid</code> and roundcube_sessauth</code> cookies persist across all three steps within an iteration, and Roundcube’s logout response invalidates the session so the next iteration starts clean. === per-worker results === worker iterations login_ok logout_ok ok? ---------------------------------------------------------------------------------- http://worker1:9091 45 45 45 PASS http://worker2:9091 45 45 45 PASS ---------------------------------------------------------------------------------- TOTAL 90 90 90 PASS</code></pre> All 90 iterations completed successfully across both workers: every login extracted a valid CSRF token, every POST received the inbox page, and every logout returned to the login screen. This validates the full extractor → template substitution → cookie management pipeline running inside the Rust FFI layer on remote machines. Deep Dive: Performance Analysis🔗</a></h2> To understand the performance gaps, I profiled loadgen-rs using perf</code> and samply</a>. The setup: # Build with debug symbols but full optimization cargo build --profile profiling # inherits release, adds debug=2 # Record CPU profile perf record -g --call-graph dwarf -o /tmp/perf-h2.data -- \ ./target/profiling/loadgen-rs --alpn-list h2 -D 10s -c 64 -t 4 -m 8 \ --insecure https://127.0.0.1:8082/?s=256k</code></pre>H2 CPU Profile🔗</a></h3> Self %</th> Function</th> Category</th></tr></thead> 27%</td> ring::aes_gcm_dec_update</code></td> TLS decrypt (AES-256-GCM)</td></tr> 21%</td> __memmove_avx</code> (libc)</td> Memory copies</td></tr> 16%</td> Kernel</td> TCP read/write syscalls</td></tr> 2.0%</td> rustls::read_opaque_message_header</code></td> TLS framing</td></tr> 1.4%</td> _mi_page_malloc</code></td> Allocator (mimalloc)</td></tr> 1.3%</td> h2::Connection::poll2</code></td> H2 protocol state machine</td></tr> 1.3%</td> h2::DynConnection::recv_frame</code></td> H2 frame processing</td></tr> 1.2%</td> BytesMut::split_to</code></td> Buffer management</td></tr> </tbody></table> ~64% of CPU time is spent outside of our code entirely — in TLS decryption, memory copies, and kernel syscalls. Only ~2.5% is in the h2 crate’s protocol logic. The H2 latency gap (1.8x vs h2load) is not a CPU bottleneck — it’s architectural. The h2 crate uses an async mpsc channel between SendRequest</code> (our task) and the connection driver task (h2’s internal I/O loop). Each request crosses this channel boundary at least twice (send + receive), adding idle time that doesn’t show up in CPU profiles. h2load, by contrast, runs everything in a single libev event loop tick — nghttp2 processes the request, writes the frame, and reads the response without any task switching. H3 CPU Profile🔗</a></h3> Self %</th> Function</th> Category</th></tr></thead> 15%</td> ring::aes_gcm_dec_update</code></td> TLS decrypt</td></tr> 13%</td> __memmove_avx</code> (libc)</td> Memory copies</td></tr> 12%</td> Kernel</td> UDP recvmsg syscalls</td></tr> 3.5%</td> quinn_proto::Connection::process_payload</code></td> QUIC packet processing</td></tr> 3.1%</td> quinn_proto::Connection::handle_decode</code></td> QUIC packet decode</td></tr> 2.9%</td> quinn::endpoint::RecvState::poll_socket</code></td> UDP socket I/O</td></tr> 2.8%</td> binary_heap::PeekMut::pop</code></td> Timer/scheduling (loss detection, CC)</td></tr> 1.8%</td> ring::aes_gcm::open</code></td> Additional TLS overhead</td></tr> 1.7%</td> quinn::RecvStream::poll_read_chunk</code></td> Stream receive</td></tr> 1.7%</td> quinn_proto::Endpoint::handle</code></td> Packet routing/demux</td></tr> 1.6%</td> h3::proto::varint::VarInt::decode</code></td> H3 framing</td></tr> </tbody></table> The QUIC protocol stack (quinn-proto) accounts for ~15% of CPU — compared to ~2.5% for h2. This is the fundamental cost of implementing QUIC in userspace Rust: Per-packet crypto: QUIC encrypts each UDP datagram individually (vs TLS record batching over TCP)</li> Congestion control state machine: Quinn maintains timer heaps (binary_heap::pop</code> at 2.8%) for loss detection and CC that TCP offloads to the kernel</li> Packet demux: Endpoint::handle</code> routes each incoming UDP datagram to the correct connection — work that the kernel’s TCP stack does for free</li> </ul> h2load uses ngtcp2 (C) and nghttp3 (C) which have tighter inner loops, less allocation overhead, and direct writev()</code> for packet batching. What the Profile Means🔗</a></h3> The profiling data tells a clear story: we’ve hit the performance ceiling of the Rust crate ecosystem, not a bug in our code. For H2, the bottleneck is TLS + memcpy + kernel — things we can’t optimize without replacing ring/rustls. For H3, quinn’s QUIC protocol processing adds ~15% overhead that ngtcp2 avoids by being a leaner C implementation. Optimization Attempts (What Worked and What Didn’t)🔗</a></h3> During development, I tried several optimizations to close the gap with h2load: What Worked🔗</a></h4> Optimization</th> Effect</th> Protocol</th></tr></thead> Raw h2 crate instead of hyper</td> Eliminated one abstraction layer</td> H2</td></tr> Raw TCP + httparse instead of hyper for H1</td> +15-25% RPS vs hyper-based H1</td> H1</td></tr> initial_rtt(1ms)</code> for quinn</td> Faster congestion control ramp-up on low-latency links</td> H3</td></tr> 1 GB stream/connection windows</td> Matches h2load’s window sizes</td> H2, H3</td></tr> H2 flow control: 512 KB batch release</td> Reduces WINDOW_UPDATE frame overhead</td> H2</td></tr> UDP socket buffers 4 MB via socket2</td> Fewer kernel drops under load</td> H3</td></tr> mimalloc global allocator</td> Better multi-threaded malloc performance</td> All</td></tr> Pre-built request templates</td> Avoid re-building HTTP requests per iteration</td> All</td></tr> Per-worker current-thread runtimes</td> No cross-thread lock contention</td> All</td></tr> </tbody></table> What Didn’t Work🔗</a></h4> Optimization</th> Expected</th> Actual</th> Why</th></tr></thead> Thread-local shared quinn::Endpoint</td> Less per-connection overhead</td> -12% RPS</td> UDP socket contention — all connections on one worker competed for the same socket</td></tr> H2 immediate flow control release (per-chunk)</td> Lower latency</td> -5.6% RPS</td> Generated too many WINDOW_UPDATE frames and cross-task messages</td></tr> Cubic congestion controller (instead of NewReno)</td> Match h2load’s ngtcp2 CC</td> -7% RPS</td> Quinn’s Cubic implementation doesn’t scale well with 512 concurrent connections on localhost</td></tr> http2 crate (0x676e67 fork with parking_lot)</td> Faster mutex in h2 hot path</td> -2.3% RPS</td> The h2 protocol logic is only 2.5% of CPU — faster locks don’t help when locks aren’t the bottleneck</td></tr> </tbody></table> The failed optimizations were valuable lessons: profiling before optimizing would have saved time. The http2 fork experiment, for instance, was motivated by the assumption that h2’s internal locking was a bottleneck — but perf showed it was only 1.3% of CPU time. Architectural Comparison: loadgen-rs vs h2load🔗</a></h3> Understanding why the gaps exist requires comparing the architectures: H2: Async Tasks vs Single Event Loop🔗</a></h4> loadgen-rs: Worker Task ──mpsc──▶ h2 Connection Driver Task ──▶ TCP h2load: libev event loop ──▶ nghttp2 ──▶ writev() ──▶ TCP</code></pre> The h2 crate splits request handling across two async tasks connected by an mpsc channel. Each request crosses this boundary twice. h2load runs everything synchronously in one event loop tick — zero context switches per request. This explains the latency gap (1.8x) without a throughput gap: the pipeline stays full, but each individual request takes longer to complete. H3: Userspace QUIC vs Lean C Implementation🔗</a></h4> loadgen-rs: quinn (Rust) ──▶ ring (Rust/asm) ──▶ UDP socket h2load: ngtcp2 (C) ──▶ OpenSSL/BoringSSL (C/asm) ──▶ UDP socket</code></pre> Both implement the same QUIC protocol, but ngtcp2 has advantages: Tighter C inner loops with less allocation</li> Direct sendmsg</code>/recvmsg</code> batching</li> Fewer abstraction layers between protocol logic and I/O</li> </ul> Quinn adds Tokio task scheduling, channel-based I/O between endpoint and connection state, and Rust’s safety overhead (bounds checks, reference counting on Bytes</code>). Deep Dive: Internal Architecture🔗</a></h2> Enum-Based Driver Dispatch🔗</a></h3> Instead of dyn Trait</code> with async boxing, I use an enum: pub enum Connection { H1Raw(h1_raw::H1RawConnection), H2(h2::H2Connection), H3(h3::H3Connection), } impl Connection { pub async fn send_request(&mut self, config: &RequestConfig) -> Result<RequestResult, RequestError> { match self { Connection::H1Raw(c) => c.send_request(config).await, Connection::H2(c) => c.send_request(config).await, Connection::H3(c) => c.send_request(config).await, } } }</code></pre> Benefits: No heap allocation per dispatch</li> Compiler can inline</li> No Pin<Box<dyn Future>></code> overhead</li> </ul> FFI Two-API Design (Batch vs Step)🔗</a></h3> The Deno integration isn’t just a thin wrapper — it’s built on a Cargo workspace with a dedicated loadgen-ffi</code> crate that compiles to a 6.4 MB shared library (libloadgen_ffi.so</code>) exposing 13 C ABI functions via Deno.dlopen()</code>. Two distinct FFI APIs🔗</a></h4> The shared library provides two API levels, each serving a different use case: 1. Batch API — for high-throughput load generation: loadgen_create(config_json) → handle loadgen_run(handle) → report_json (nonblocking in Deno) loadgen_metrics_snapshot(handle) → json loadgen_destroy(handle)</code></pre> This wraps the existing run_from_config()</code> engine — the same optimized H1/H2/H3 drivers, per-worker runtimes, and HdrHistograms that match h2load performance. A single loadgen_run</code> call spawns the full benchmark and returns a complete RunReport</code> as JSON. 2. Step API — for scripted request sequences with correlation: loadgen_step_session_create(session_config_json) → session_handle loadgen_step_execute(session_handle, request_json) → response_json (nonblocking) loadgen_step_session_reset(session_handle) loadgen_step_session_destroy(session_handle)</code></pre> The Step API uses reqwest as its HTTP client rather than the custom drivers. This is a deliberate architectural choice: reqwest provides built-in cookie jars, redirect following, H1/H2/H3 support, and connection pooling — features that the raw benchmark drivers intentionally omit for performance. In script mode, correctness (cookies, redirects, body capture) matters more than raw throughput. Panic-safe FFI boundary🔗</a></h4> Every exported function is wrapped in std::panic::catch_unwind()</code> to prevent Rust panics from unwinding across the C ABI boundary (which would be undefined behavior). Errors are stored in a thread-safe slot (OnceLock<Mutex<Option<String>>></code>) and retrieved via loadgen_last_error()</code>. All returned char*</code> strings must be freed with loadgen_free_string()</code> — the contract is documented in docs/ffi-contract.md</code>. TypeScript script engine🔗</a></h4> On the Deno side, runScriptScenario()</code> in ts/script_mode.ts</code> implements a k6-inspired scenario runner with: VU parallelism: N virtual users run as concurrent Promise.all</code> tasks</li> Duration-based execution: VUs loop until the target duration expires</li> Two execution modes: fetch</code> (Deno’s native HTTP client) and ffi-step</code> (Rust reqwest via the Step API)</li> Template engine: {{variable}}</code> substitution in URLs, headers, and bodies</li> Extractor system: Capture values from JSON paths, response headers, regex groups, or DOM CSS selectors (via deno-dom) — stored in per-VU state for use in subsequent steps</li> Check system: Eight check types including status_in</code>, body_includes</code>, header_exists</code>, header_equals</code>, header_includes</code>, json_path_exists</code>, json_path_equals</code>, and regex_match</code></li> Cookie jar: Full RFC-aware implementation in TypeScript (for fetch</code> mode) with domain/path matching, expiry handling, and Secure attribute support</li> </ul> The ffi-step</code> mode delegates cookie management and redirects to reqwest on the Rust side, while the fetch</code> mode handles everything in TypeScript. Both modes share the same check and extractor evaluation logic. The full scripting documentation is in loadgen-script</a> Example coverage🔗</a></h4> The examples/</code> directory in loadgen-rs repository</a> demonstrates the full feature surface: Example</th> What it demonstrates</th></tr></thead> simple.ts</code></td> Minimal FFI benchmark (10 requests, single client)</td></tr> correlation_mvp.ts</code></td> Login, extract JWT token, use in follow-up request</td></tr> cookie_redirect_local_mvp.ts</code></td> Embedded Deno server with Set-Cookie + 302 redirect chain</td></tr> protocol-matrix.ts</code></td> H1/H2/H3 batch + script mode for all three protocols</td></tr> ab-stages.ts</code></td> Ramping VUs with stage interpolation (k6-like stages)</td></tr> native_step_skeleton.ts</code></td> Direct Step API usage without the scenario runner</td></tr> all-functionality.ts</code></td> Full integration test suite using Chai assertions</td></tr> </tbody></table> The cookie_redirect_local_mvp.ts</code> example is particularly interesting: it spins up a local Deno HTTP server that sets cookies and issues redirects, then runs the script scenario against it — a self-contained integration test that validates the entire cookie + redirect + check pipeline. Appendix: Commands and Reproducibility🔗</a></h2> Equivalent Command Pairs (h2load vs loadgen-rs)🔗</a></h3> To compare both tools fairly, use equivalent flags and remember: --rps</code> is per client in both tools. # 1) H1 count mode (same request budget) h2load --h1 -n 200 -c 10 -t 8 -m 1 "https://bench.local:8082/?s=256k" loadgen-rs --h1 -n 200 -c 10 -t 8 -m 1 "https://bench.local:8082/?s=256k" # 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients * 20 rps/client) h2load --h1 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --h1 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" # 3) H2 duration mode with pacing (same effective target) h2load --alpn-list=h2 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --alpn-list h2 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" # 4) H3 duration mode with pacing (same effective target) h2load --alpn-list=h3 -D 10s -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k" loadgen-rs --alpn-list h3 --duration 10s -n 0 -c 10 -t 8 -m 1 --rps 20 "https://bench.local:8082/?s=256k"</code></pre>CLI Compatibility Map🔗</a></h3> The table below maps loadgen-rs</code> options to their closest h2load</code> equivalent. loadgen-rs</code></th> h2load</code></th> Mapping</th> Notes</th></tr></thead> <URL></code></td> <URI>...</code></td> partial</td> h2load</code> accepts multiple URIs</td></tr> -n <REQUESTS></code></td> -n, --requests=<N></code></td> same</td> </td></tr> -D, --duration</code></td> -D, --duration</code></td> same</td> </td></tr> --warm-up-time</code></td> --warm-up-time</code></td> same</td> </td></tr> --ramp-up-time</code></td> none</code></td> no direct equivalent</td> Closest knobs are -r/--rate</code> + --rate-period</code>, but semantics differ</td></tr> -c <CLIENTS></code></td> -c, --clients=<N></code></td> same</td> </td></tr> -t <THREADS></code></td> -t, --threads=<N></code></td> same</td> </td></tr> -m <MAX_STREAMS></code></td> -m, --max-concurrent-streams=<N></code></td> same</td> </td></tr> --h1</code></td> --h1</code></td> same</td> </td></tr> --alpn-list h2|h3</code></td> --alpn-list=<LIST></code></td> partial</td> h2load</code> allows arbitrary ALPN lists</td></tr> --connect-timeout</code></td> none</code></td> no direct equivalent</td> Only related connection-level timeout controls exist (-T</code>, -N</code>)</td></tr> --request-timeout</code></td> none</code></td> no direct equivalent</td> No per-request timeout flag in h2load</code></td></tr> --tcp-quickack</code></td> none</code></td> none</td> </td></tr> --method <METHOD></code></td> none</code></td> no direct equivalent</td> h2load</code> is primarily GET; POST</code> is enabled via -d</code></td></tr> -H, --header</code></td> -H, --header</code></td> same</td> </td></tr> -d, --data <DATA></code></td> -d, --data=<PATH></code></td> partial</td> loadgen-rs</code>: inline body; h2load</code>: file path</td></tr> --data-file <PATH></code></td> -d, --data=<PATH></code></td> close equivalent</td> </td></tr> --rps <RPS></code></td> --rps=<N></code></td> same</td> per-client target in both tools (aggregate RPS * clients</code>)</td></tr> -k, --insecure</code></td> none</code></td> no CLI equivalent</td> No direct insecure toggle in h2load</code> CLI</td></tr> --tls-ciphers</code></td> --ciphers</code>, --tls13-ciphers</code></td> partial</td> h2load</code> splits TLS <=1.2 and TLS 1.3 ciphers</td></tr> --tls-ca <PATH></code></td> none</code></td> none</td> </td></tr> -o, --output <PATH></code></td> none</code></td> none</td> h2load</code> offers --log-file</code> (per-request TSV), not run-level JSON/CSV</td></tr> --format jsonl|csv</code></td> none</code></td> none</td> </td></tr> --no-human</code></td> none</code></td> none</td> </td></tr> --tail-friendly</code></td> none</code></td> none</td> </td></tr> --metrics-sample</code></td> none</code></td> none</td> </td></tr> </tbody></table> Container Baseline🔗</a></h3> The multi-stage Containerfile builds the binary in a Rust image and copies it into a minimal Debian Slim image with bash</code> and curl</code>, with HTTP/3 support built in: podman build -t loadgen-rs -f Containerfile . podman run --rm --network host \ --cap-add=SYS_ADMIN \ --security-opt seccomp=unconfined \ loadgen-rs -D 10s -c 8 -t 4 --alpn-list=h2 https://target:8443/</code></pre> --network host</code> avoids NAT overhead. --cap-add=SYS_ADMIN</code> enables io_uring inside the container. Test Server Setup🔗</a></h3> The test server used in this post is haterm</code> from HAProxy</code>, created and started with the following commands: # create combined.pem in TLS-PATH cat server.crt ca.crt server.key > combined.pem # build container image buildah bud -f Containerfile-hap-git -t hap-own # run haterm podman run -d --rm --name hap-own \ --network host \ --entrypoint=/usr/local/sbin/haterm \ -v <TLS-PATH>/tls:/mnt:ro \ localhost/haterm:latest \ -L 127.0.0.1:8081 \ -F 'bind quic4@127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h3' \ -F 'bind 127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h2'</code></pre> The used Containerfile-hap-git</a> Cloud Deployment (Terraform + Ansible)🔗</a></h3> Running distributed benchmarks on your local machines is useful for development, but real load testing demands dedicated cloud infrastructure — consistent hardware, dedicated NICs, and geographic proximity to the target. loadgen-rs ships with a complete Terraform + Ansible pipeline for Hetzner Cloud that takes you from zero to running workers in under five minutes. The Terraform configuration provisions dedicated-vCPU instances (CCX series), places them in a private network, and configures a firewall that allows SSH and the worker-agent port. A single generate-inventory.sh</code> script bridges the gap between Terraform and Ansible — it reads the Terraform outputs and writes a ready-to-use ansible/inventory.ini</code> with the public IPs: terraform apply │ ├── hcloud_server.worker[0] → 49.13.x.x (loadgen-worker-0) ├── hcloud_server.worker[1] → 49.13.y.y (loadgen-worker-1) ├── hcloud_network → 10.0.1.0/24 (private) └── hcloud_firewall → SSH + port 9091 │ ▼ ./generate-inventory.sh │ ▼ ansible/inventory.ini ├── loadgen-worker-0 ansible_host=49.13.x.x worker_port=9091 └── loadgen-worker-1 ansible_host=49.13.y.y worker_port=9091 │ ▼ ansible-playbook deploy-workers.yml │ ▼ Workers ready — POST /run from controller</code></pre> The full workflow from provisioning to results: # 1. Provision Hetzner Cloud machines cd terraform cp terraform.tfvars.example terraform.tfvars # Set: hcloud_token, ssh_key_name, worker_count=4, worker_type=ccx33 terraform init && terraform apply # 2. Generate Ansible inventory from Terraform state ./generate-inventory.sh # → ansible/inventory.ini with 4 workers # 3. Deploy + benchmark + cleanup in one command cd .. scripts/distributed-remote.sh \ -i ansible/inventory.ini \ --target-url "https://bench.target:8443/?s=256k" \ -c 1000 -D 60 --stop-after # 4. Tear down when done cd terraform && terraform destroy</code></pre> The Ansible playbook (deploy-workers.yml</code>) handles everything on the remote machines: installing Deno if not present, copying the FFI shared library and TypeScript files, deploying a systemd service unit (with a tmux fallback for systems without systemd), and running health checks to confirm each worker is ready. The matching stop-workers.yml</code> playbook cleanly shuts everything down. This separation of concerns — Terraform owns infrastructure, Ansible owns configuration, the shell script orchestrates the workflow — means you can mix and match. Already have machines? Skip Terraform, write inventory.ini</code> by hand. Prefer a different cloud? Replace the .tf</code> files, keep the same Ansible playbooks and scripts. The worker-agent doesn’t care how it got deployed. Component</th> Responsibility</th></tr></thead> terraform/*.tf</code></td> Provision Hetzner VMs, network, firewall</td></tr> terraform/generate-inventory.sh</code></td> Bridge: Terraform outputs → Ansible inventory</td></tr> ansible/deploy-workers.yml</code></td> Install Deno, deploy files, start workers</td></tr> ansible/stop-workers.yml</code></td> Stop workers (systemd or tmux)</td></tr> scripts/distributed-remote.sh</code></td> End-to-end: deploy → benchmark → stop</td></tr> </tbody></table> Conclusion🔗</a></h2> loadgen-rs is a lean, focused tool: it performs exactly one run, outputs the results as JSONL, and leaves the orchestration (warmup, parameter sweeps, plotting) to the calling script. The combination of raw h2, quinn, and rustls covers all three HTTP protocols — and thanks to per-worker runtimes and a lock-free design, the client itself is not the bottleneck. CPU profiling confirmed this: over 60% of execution time is spent in TLS decryption, memory copies, and kernel syscalls — code paths shared by any HTTP benchmark client regardless of language. The remaining gaps to h2load are architectural properties of the Rust crate ecosystem (h2’s async channel design, quinn’s userspace QUIC overhead) rather than optimization opportunities in loadgen-rs itself. With the Deno FFI layer, distributed mode, and Terraform/Ansible pipeline, loadgen-rs now serves five usage patterns: shell scripts that need a fast, h2load-compatible CLI tool; TypeScript users who want k6-style scripted scenarios with native H1/H2/H3 performance; multi-machine deployments where a single process can’t generate enough load; cloud-provisioned benchmarks where you spin up dedicated Hetzner instances, run the test, and tear everything down in minutes; and distributed scripted scenarios where workers execute multi-step request sequences with checks and response validation on remote machines. The two-API design (batch for throughput, step for correlation) plus the controller/worker/infrastructure stack bridges all five without compromising any of them. Code and Repository Link🔗</a></h3> The code is on loadgen-rs - GitHub</a> — PRs welcome. cert-manager-webhook-libdns: One Webhook, Many DNS Providers 2026-03-03T00:00:00+00:00 cert-manager-webhook-libdns: One Webhook, Many DNS Providers🔗</a></h1> Managing DNS-01 challenges with cert-manager often means choosing one provider-specific integration and staying tightly coupled to it. cert-manager-webhook-libdns</a> takes a different approach: it uses the libdns</a> ecosystem so one webhook can support many DNS providers through a shared interface. If you run mixed infrastructure, migrate between DNS vendors, or just want a cleaner operational model, this project is built for that. Why this project exists🔗</a></h2> In many Kubernetes setups, certificate automation becomes harder than it should be because DNS integrations are fragmented: Every provider has different APIs, credentials, and operational quirks.</li> Switching providers often means switching tooling.</li> Testing and maintenance can become provider-specific and brittle.</li> </ul> cert-manager-webhook-libdns</a> solves this by exposing a single cert-manager webhook that delegates DNS record operations to pluggable libdns providers. What it does🔗</a></h2> At a high level, the webhook: Receives DNS-01 challenge requests from cert-manager.</li> Loads provider credentials from Kubernetes Secrets.</li> Resolves and writes challenge TXT records through the selected libdns provider.</li> Cleans up records after validation.</li> </ul> It supports important real-world behaviors, including: Handling multiple TXT values for wildcard + apex certificate requests.</li> Safer fallback logic when record listing fails.</li> Provider-specific TTL handling (for example, deSEC minimum TTL enforcement).</li> Deterministic provider listing in CLI output.</li> </ul> Operational improvements in recent iterations🔗</a></h2> Several practical improvements were added to make this project easier to run and maintain: Semantic version output via --version</code> (currently v0.5.0</code>).</li> No etcd/envtest dependency in the default make test</code> path.</li> Cleaner container workflow with Containerfile</code> support.</li> Release workflow hardening:</li> tests run before image release</li> latest</code> is only published for stable tags</li> release creation happens after successful image push</li> concurrency guard to prevent overlapping tag releases</li> </ul> These are the kinds of changes that reduce surprises in CI/CD and day-to-day operations. Provider compatibility is now automated🔗</a></h2> A major maintenance challenge with libdns-based projects is knowing which upstream provider modules are actually compatible with your pinned libdns</code> version. This repository now includes an automated compatibility process: Script: scripts/check_libdns_provider_compat.sh</code></li> Workflow: .github/workflows/libdns-provider-compat.yml</code></li> Report snapshot: docs/01_provider-compat-latest.md</code></li> Historical snapshots: docs/YYYY-MM-DD_provider-compat.md</code></li> </ul> The check uses gh api</code> to scan provider repositories, inspect their go.mod</code>, and classify compatibility against the libdns version used in this project. That gives maintainers a repeatable compatibility signal instead of manual guesswork. Quick start🔗</a></h2> 1. Build and publish the image🔗</a></h3> Use your preferred container runtime (Podman is preferred, Docker fallback is supported): make container-build make container-push IMAGE_TAG=v0.5.0 REGISTRY=ghcr.io/<your-org></code></pre>2. Deploy the Helm chart🔗</a></h3> helm install libdns-webhook ./deploy/libdns-webhook \ --namespace cert-manager \ --set image.repository=ghcr.io/<your-org>/cert-manager-webhook-libdns \ --set image.tag=v0.5.0 \ --set groupName=acme.yourdomain.com</code></pre>3. Create provider credentials secret🔗</a></h3> Example for token-based providers: apiVersion: v1 kind: Secret metadata: name: dns-provider-credentials namespace: cert-manager type: Opaque stringData: api_token: "<YOUR_API_TOKEN>"</code></pre>4. Configure ClusterIssuer🔗</a></h3> apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: your-email@example.com privateKeySecretRef: name: letsencrypt-prod-account solvers: - dns01: webhook: groupName: acme.yourdomain.com solverName: libdns config: provider: cloudflare secretRef: name: dns-provider-credentials namespace: cert-manager</code></pre>Useful CLI checks🔗</a></h2> Before deployment, you can quickly inspect the binary: ./webhook --version ./webhook --list-providers</code></pre> This is helpful for validating exactly what was compiled into your image. Who should use this🔗</a></h2> This project is a strong fit if you: Run cert-manager and use DNS-01 challenges.</li> Need flexibility across multiple DNS providers.</li> Want one consistent webhook integration instead of many provider-specific implementations.</li> Care about release discipline and compatibility visibility.</li> </ul> Final thoughts🔗</a></h2> cert-manager-webhook-libdns</a> is not just a DNS integration layer. It is an operational strategy: keep cert-manager integration stable while letting providers vary underneath. If your platform team wants fewer moving parts and more predictable certificate automation, this is a practical approach worth adopting. Simple autotls setup with ferron 2025-10-22T00:00:00+00:00 Introduction🔗</a></h1> ferron</a> is a webserver written in rust. This webserver have the Automatic TLS</a> feature for deSEC</a> implemented which makes it quite easy to create a TLS http server listen on localhost 🤩 🥳. Pre-Requirements: curl</a></li> deSEC Account</a></li> own Domain => referred in this post as DOMAIN</code></li> </ul> Get Token.🔗</a></h1> To be able to add the acme challenge to the deSEC is a Token required. How to get and mange tokens for deSEC is documented at Manage Tokens</a> config and start ferron🔗</a></h1> The config file ferron.kdl</code>. * { protocols "h1" "h2" "h3" log "/dev/stdout" error_log "/dev/stderr" default_http_port 8080 // for real webserver change the port to 80 default_https_port 8443 // for real webserver change the port to 443 trust_x_forwarded_for #true auto_tls auto_tls_contact "Your-letsencrypt@Email-address" auto_tls_cache "letsencrypt-cache" auto_tls_letsencrypt_production auto_tls_challenge "dns-01" provider="desec" api_token="THE-TOKEN" } *.DOMAIN { status 200 url="/static_return" body="OK" root "wwwroot" }</code></pre> After the Install</a> process can you use the above config file to start ferron. # ./ferron -c /datadisk/Downloads/ferronweb/ferron-2.0.0-beta.16/ferron.kdl HTTP server is listening on [::]:8080... HTTPS server is listening on [::]:8443... HTTP/3 server is listening on [::]:8443...</code></pre> tip </div> tip </div> Wait at least ~2 minutes before the first request because the deSEC API and DNS Server needs some time to propagate the acme challenge. </div> </div> Finally🔗</a></h1> Wenn everything works as designed should can you cal curl on localhost with an valid TLS Certificate. curl -v --compressed \ --resolve test2.DOMAIN:8443:127.0.0.1 \ --http2 --http2-prior-knowledge \ -H 'X-Forwarded-For: 10.1.1.1,20.2.2.2' \ https://test2.DOMAIN:8443/static_return </code></pre> ferron shows then the right IP from the first X-Forwarded-For</code> entry. # ./ferron -c /datadisk/Downloads/ferronweb/ferron-2.0.0-beta.16/ferron.kdl HTTP server is listening on [::]:8080... HTTPS server is listening on [::]:8443... HTTP/3 server is listening on [::]:8443... 10.1.1.1 - - [22/Oct/2025:23:08:02 +0200] "GET /static_return HTTP/2.0" 200 2 "-" "curl/8.5.0"</code></pre> How to automate TLSA update 2025-09-28T00:00:00+00:00 Introduction🔗</a></h1> “DNS-based Authentication of Named Entities (DANE)” is specified in rfc6698</a> and it requires DNSSEC (DANE only works when DNSSEC is enabled). DNSSEC is defined in RFC4033</a>, RFC4034</a>, and RFC4035</a>. I describe here my process how I updated auotmatically the TLSA record for my mailserver via the deSEC rrsets API</a>. If you like please be so kind and Donate</a> to deSEC Pre-Requirements: jq</a></li> curl</a></li> certbot</a></li> deSEC Account</a></li> own Domain</li> </ul> DeSEC, domain + mail setup🔗</a></h1> The “Mail setup for deSEC” is described at the link Mail setup for deSEC</a>. certbot🔗</a></h1> Certbot offers the option to run some hooks at certificate handling Pre and Post Validation Hooks</a>. json Payload🔗</a></h2> The file tlsa-update.json</code> have this content. [ { "subname": "_25._tcp.mail", "type": "TLSA", "ttl": 3600, "records": [ "3 1 1 ${TLSA_SHA}" ] } ]</code></pre>post-hook🔗</a></h2> At the end add the following script into the post hook directory /etc/letsencrypt/renewal-hooks/post/</code>. I called it reload-services.sh</code> #!/bin/bash ## Update TLSA record export DESEC_TOKEN=<YOUR_DESEC_TOKEN> export DOMAIN_NAME=<YOUR_DOMAIN> export CERT_DOMAIN=<CERT_DOMAIN>.${DOMAIN_NAME} export TLSA_SHA=$(/usr/bin/openssl x509 -in /etc/letsencrypt/live/${DOMAIN}/fullchain.pem -noout -pubkey \ | /usr/bin/openssl pkey -pubin -outform DER \ | /usr/bin/openssl sha256 2>&1\ | /usr/bin/cut -f2 -d ' ') envsubst < tlsa-update.json \ | curl -sSLX PUT https://desec.io/api/v1/domains/${DOMAIN_NAME}/rrsets/ \ --header "Authorization: Token ${DESEC_TOKEN}" \ --header "Content-Type: application/json" \ --data @- \ | jq /usr/bin/systemctl restart nginx.service /usr/bin/systemctl restart dovecot.service /usr/bin/systemctl restart postfix.service</code></pre> I know that there is the variable CERTBOT_DOMAIN</code> and it could also be used in the script above, if it fit’s for your use-case Finally🔗</a></h1> Wenn everything works as designed should you never need to update the TLSA record when the LE certificate was changed. How to Run Tor Arti as an HTTP Proxy or Behind a g3 HTTP Proxy 2025-09-01T00:00:00+00:00 Introduction🔗</a></h1> This is a short guide on how to use Tor</a> through an HTTP proxy. The setup uses three components: arti</a></li> g3proxy</a></li> Any software that supports an HTTP proxy</li> </ul> The diagram below shows the overall flow. --- config: look: handDrawn theme: neutral --- flowchart LR A[Client] -->|HTTP-protocol| B[g3] -->|SOCKS-protocol| C[arti] --> D[TOR-Network] --> G[Original Destination] E[Client] -->|HTTP-protocol| B[g3] F[Client] -->|HTTP-protocol| C[arti] D[TOR-Network] --> H[Original Destination] D[TOR-Network] --> I[Original Destination] J[Client] -->|SOCKS-protocol| C[arti] </pre> Arti🔗</a></h2> Quote from the Arti homepage</a>: Arti is a complete rewrite of the C Tor codebase, and it is currently under active development. It is written in Rust, and it is designed to be modular, reusable, and easy to audit. </blockquote> You need to build Arti before you can run it 😄. I am using an Ubuntu-based distribution, so this guide uses apt</code>. This is a short summary of the original Compiling Arti</a> guide. # clone the repo git clone https://gitlab.torproject.org/tpo/core/arti.git # navigate to the directory cd arti ## install some necessary packages sudo apt install libsqlite3-dev cargo build -p arti --release</code></pre> tip </div> tip </div> You may need to run the build/install process more than once while adding missing dependencies on your build host. </div> </div> g3proxy🔗</a></h2> You can either download</a> prebuilt g3proxy packages or build it yourself. I chose to build it. git clone https://github.com/bytedance/g3 cd g3 sudo apt install liblua5.4-dev capnproto libc-ares-dev cargo build</code></pre> tip </div> tip </div> You may need to run the build/install process more than once while adding missing dependencies on your build host. </div> </div> g3proxy has many more configuration options than shown here. See the project README</a>. Config🔗</a></h3> g3proxy requires a configuration file. --- runtime: thread_number: 2 log: stdout server: - name: http_proxy escaper: default type: http_proxy listen: address: "[::]:10087" # <<<< http-proxy listening port tls_client: { } # Open layer-7 https forward forwarding support escaper: - name: default type: proxy_socks5 proxy_addr: "[::1]:9150" # <<<< arti listening port</code></pre>Runtime🔗</a></h2> After both tools (Arti and g3proxy) are built, open two shells: run Arti in one shell and g3proxy in the other. Shell 1: Arti🔗</a></h3> /datadisk/git-repos/arti $ # target/release/arti proxy 2025-09-01T15:11:17Z INFO arti::subcommands::proxy: Starting Arti 1.5.0 in SOCKS proxy mode on localhost port 9150 ... 2025-09-01T15:11:17Z INFO tor_memquota::mtracker: Memory quota tracking initialised max=8.00 GiB low_water=6.00 GiB 2025-09-01T15:11:17Z INFO arti_client::client: Using keystore from "/home/alex/.local/share/arti/keystore" 2025-09-01T15:11:17Z INFO tor_dirmgr: Marked consensus usable. 2025-09-01T15:11:17Z INFO tor_dirmgr: Loaded a good directory from cache. 2025-09-01T15:11:17Z INFO arti::subcommands::proxy: Sufficiently bootstrapped; system SOCKS now functional. 2025-09-01T15:11:17Z INFO arti::socks: Listening on [::1]:9150. 2025-09-01T15:11:17Z INFO arti::socks: Listening on 127.0.0.1:9150. 2025-09-01T15:11:17Z INFO tor_dirmgr: Marked consensus usable. 2025-09-01T15:11:17Z INFO tor_dirmgr: Directory is complete. attempt=1 2025-09-01T15:11:17Z INFO tor_guardmgr::guard: We have found that guard [scrubbed] is usable. 2025-09-01T15:11:18Z INFO arti::reload_cfg: Successfully reloaded configuration. 2025-09-01T15:11:28Z INFO tor_guardmgr::guard: We have found that guard [scrubbed] is usable.</code></pre>Shell 2: g3proxy🔗</a></h3> /datadisk/git-repos/g3 $ # target/debug/g3proxy --config-file g3proxy/examples/chain_socks_proxy/g3proxy.yaml</code></pre> Now configure your browser to use the HTTP proxy at 127.0.0.1:10087</code>, then test with curl</code>. --- config: theme: neutral --- flowchart LR A[`curl`] -->|`127.0.0.1:10087`| B[g3] -->|`SOCKS`| C[arti] --> D[TOR-Network] -->|`https`| G[https://check.torproject.org/] </pre> Shell 3🔗</a></h3> # curl -sSL \ --proxy 127.0.0.1:10087 \ https://check.torproject.org/ \ |egrep -i cong Congratulations. This browser is configured to use Tor. Congratulations. This browser is configured to use Tor.</code></pre>Some Thoughts🔗</a></h2> Because curl</code> also supports SOCKS, you can point curl</code> directly to Arti. However, many applications support only HTTP proxying and not SOCKS. For those applications, this setup is useful. There are also other tools for HTTP(S)-to-SOCKS translation, for example Privoxy forward-socks5</code></a>. I chose g3 because it is relatively small and written in Rust 😄. Update🔗</a></h2> Since Arti 1.7.0 (October 30, 2025)</a>, HTTP CONNECT</code> is supported. That means Arti can be used as an HTTP proxy endpoint, not only as a SOCKS5 endpoint. How to Host a Zola-Generated Site on GitLab Pages 2025-08-26T00:00:00+00:00 Introduction🔗</a></h1> After some time without a personal blog, I decided to write down what I have learned and used. This site is mainly a personal collection of tasks, tools, and setups that I use regularly. Over the years, I tried plain text, plain HTML, and several CMSes and frameworks such as WordPress, Django, Gatsby, and Hugo. For some company websites, I still use a CMS because many business users are not comfortable with Git and text editors, and a CMS offers a more user-friendly interface. For my personal site, I chose Zola</a> because I wanted a Rust-based static site generator. When I checked Jamstack Site Generators</a>, I found a few Rust options 🦀 and picked Zola. Why GitLab Pages?🔗</a></h2> These days, hosting a static site for free is straightforward. Many version control systems offer a Pages feature, including GitLab Pages</a>. Why GitLab and not GitHub, Codeberg, or something else? Simply put: I already had a GitLab account and I like the platform. What do I publish?🔗</a></h2> After choosing the tooling, the next question was content. There are already many blogs out there 🤭 on almost every topic, and more people now use AI to get answers quickly. My honest answer is: “for my own laziness” 🤓. I had knowledge spread across multiple places, so I kept searching for where I had documented a particular command or setup. This blog is my way to keep that knowledge in one place that I can access from anywhere. Summary🔗</a></h2> So far, the setup looks like this: Tool: zola</li> Hosting: GitLab Pages</li> Content: Notes for myself, and hopefully useful for others too</li> </ul> Let’s start writing🔗</a></h2> zola🔗</a></h3> The Zola docs begin with an overview</a> and the core commands: zola init none-blog cd none-blog git remote add origin https://gitlab.com/aleks001/none-blog.git # original theme (until March 2026) # git submodule add https://codeberg.org/salif/linkita.git themes/linkita # theme used March-July 2026 # git submodule add https://github.com/thomasweitzel/zolarwind.git themes/zolarwind # current theme (since July 2026) git submodule add --branch v4 https://codeberg.org/salif/linkita.git themes/linkita</code></pre> The last command adds the Linkita</a> theme, which is what this site currently runs on. Update (March 2026): I moved from Linkita to Zolarwind</a> because I preferred the design. The commented command above reflects my original setup when I first published this post. Update (July 2026): I moved back to Linkita</a>. I started publishing some posts in German alongside English, and Zolarwind was never built with multilingual support in mind — it hardcodes the site’s default language in several templates. Linkita has native per-page language handling, a working language switcher, and per-language taxonomies built in, so this time the switch was about capability, not just visual preference. After that, I could start writing. You can find my blog posts here</a>, and the list will keep growing. The site config is here</a>. Create Content🔗</a></h3> Based on what I learned from analytics tools, I always include front matter like this: --- title: How to host a Zola-generated site on GitLab Pages description: How to host a Zola-generated site on GitLab Pages updated: 2025-08-26 date: 2025-08-26 taxonomies: categories: - zola - gitlab tags: - zola - gitlab - rust ---</code></pre> Then I just create Markdown files and write. GitLab Pages🔗</a></h3> To use GitLab Pages</a>, you need a GitLab account and a repository. This is mine for this blog: https://gitlab.com/aleks001/none-blog/</a>. Zola provides a GitLab Pages deployment guide</a>. I adapted my .gitlab-ci.yml</code> based on that guide. I also added a compression step so content is served compressed to browsers, which saves bandwidth. This is my .gitlab-ci.yml</code>: stages: - build - deploy default: image: debian:stable-slim variables: # Fetch theme submodules automatically. GIT_SUBMODULE_STRATEGY: "recursive" # Keep CI on the same major/minor Zola as local development. ZOLA_VERSION: "0.22.1" PANDOC_VERSION: "3.9" TYPST_VERSION: "0.15.0" SITE_BASE_URL: "https://blog.none.at" COMPRESS_REGEX: ".*\\.(htm|html|xml|txt|text|js|css|svg|pdf)$" build-pdfs: stage: build script: - | apt-get update DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends \ wget ca-certificates python3 python3-yaml xz-utils \ fonts-lato fonts-hack echo "* Installing pandoc ${PANDOC_VERSION}" PANDOC_TARBALL="pandoc-${PANDOC_VERSION}-linux-amd64.tar.gz" wget --no-verbose "https://github.com/jgm/pandoc/releases/download/${PANDOC_VERSION}/${PANDOC_TARBALL}" tar -xzf "${PANDOC_TARBALL}" --strip-components=2 -C /usr/local/bin pandoc-${PANDOC_VERSION}/bin/pandoc echo "* Installing typst ${TYPST_VERSION}" TYPST_TARBALL="typst-x86_64-unknown-linux-musl.tar.xz" wget --no-verbose "https://github.com/typst/typst/releases/download/v${TYPST_VERSION}/${TYPST_TARBALL}" tar -xJf "${TYPST_TARBALL}" --strip-components=1 -C /usr/local/bin \ "typst-x86_64-unknown-linux-musl/typst" echo "* Building PDFs" bash scripts/build-pdfs.sh artifacts: paths: - static/pdf expire_in: 1 week pages-build: stage: build needs: - build-pdfs script: - | apt-get update DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends wget ca-certificates python3 ZOLA_RELEASE_DIR="https://github.com/getzola/zola/releases/download/v${ZOLA_VERSION}" ZOLA_TARBALL="zola-v${ZOLA_VERSION}-x86_64-unknown-linux-gnu.tar.gz" echo "******************************************************" echo "*" echo "* Using zola version: ${ZOLA_VERSION}" echo "* Retrieving zola tarball: ${ZOLA_TARBALL}" echo "* ..from: ${ZOLA_RELEASE_DIR}" echo "*" echo "******************************************************" wget --no-verbose "${ZOLA_RELEASE_DIR}/${ZOLA_TARBALL}" tar -xzf "${ZOLA_TARBALL}" ./zola build --base-url "${SITE_BASE_URL}" --minify python3 scripts/postprocess_jsonld_redirects.py public artifacts: paths: - public expire_in: 1 week pages: stage: deploy pages: true dependencies: - pages-build script: - | echo "Deploy pages built in previous stage" apt-get update DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends brotli zstd find public -type f -regex "${COMPRESS_REGEX}" -exec gzip -f -k {} \; find public -type f -regex "${COMPRESS_REGEX}" -exec brotli -f -k {} \; find public -type f -regex "${COMPRESS_REGEX}" -exec zstd -f -k {} \; artifacts: paths: - public expire_in: 1 week rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</code></pre>Custom domains🔗</a></h3> Because GitLab Pages supports custom domains</a>, I configured the blog</code> subdomain for none.at</code>. That gives me a cleaner URL, as described in the GitLab Pages DNS records documentation</a>. Make it public🔗</a></h2> After these steps, I run the usual Git commands to publish changes: git add .</li> git commit -s -m ‘Init’</li> git push</li> </ul> I repeat this workflow for future posts with a proper commit message 😄. Mail setup for deSEC 2025-07-18T00:00:00+00:00 Introduction🔗</a></h1> This post describes how to configure the mail DNS setup for deSEC</a> via commandline. Prepare🔗</a></h1> General information🔗</a></h2> It’s recommended to install the certbot-dns-desec</a> because this makes the interaction with let’s encrypt (also written as “LE”) much easier. In this post and for the setup are the following tools required bash</code>, curl</code>, jq</code>, envsubst</code> and certbot</code>. For the beginning will we export the two environment variables below. export DNSSEC_TOKEN='Your-DNSSEC-API-Token' export DOMAIN_NAME='Your-Domain'</code></pre> The value for the DNSSEC_TOKEN</code> can be created via the Web UI or via API as documented at Creating a Token</a>. Create Certificate🔗</a></h2> The creation of the file /etc/letsencrypt/secrets/$DOMAIN_NAME.ini</code> is documented on the certbot-dns-desec</a> We can now create the LE certificate and key after the preparation above. warning </div> Warning </div> Please be aware that an wildcard certificate will be created with the call below! </div> </div> certbot certonly \ --authenticator dns-desec \ --dns-desec-credentials /etc/letsencrypt/secrets/$DOMAIN_NAME.ini \ -d '$DOMAIN_NAME' \ -d '*.$DOMAIN_NAME'</code></pre> There must be an directory /etc/letsencrypt/live/${DOMAIN_NAME}/fullchain.pem</code> after the successfully execution of the certbot call. The deSEC json file🔗</a></h2> To be able to create the DNS Resource Record Sets (RRsets) is a json file required. The json file below can be used to create the RRsets via the API call, which is shown at the end. Save the json text into the file mail-setup.json</code>. [ { "subname": "", "type": "MX", "ttl": 3600, "records": [ "5 mail.${DOMAIN_NAME}." ] }, { "subname": "", "type": "TXT", "ttl": 3600, "records": [ "\"v=spf1 mx -all\"" ] }, { "subname": "", "type": "A", "ttl": 3600, "records": [ "IPv4 Adress" ] }, { "subname": "", "type": "AAAA", "ttl": 3600, "records": [ ["IPv6 Adress"] ] }, { "subname": "", "type": "CAA", "ttl": 3600, "records": [ "0 iodef mailto:postmaster@${DOMAIN_NAME}", "0 issue letsencrypt.org" ] }, { "subname": "_25._tcp.mail", "type": "TLSA", "ttl": 3600, "records": [ "3 1 1 ${TLSA_SHA}" ] }, { "subname": "_dmarc", "type": "TXT", "ttl": 3600, "records": [ "\"v=DMARC1;p=quarantine;pct=100;rua=mailto:postmaster@${DOMAIN_NAME};ruf=mailto:postmaster@${DOMAIN_NAME}\"" ] }, { "subname": "_mta-sts", "type": "TXT", "ttl": 3600, "records": [ "\"v=STSv1; id=2023042102;\"" ] }, { "subname": "_smtp._tls", "type": "TXT", "ttl": 3600, "records": [ "\"v=TLSRPTv1; rua=mailto:postmaster@${DOMAIN_NAME}\"" ] }, { "subname": "dkim._domainkey", "type": "TXT", "ttl": 3600, "records": [ "\"v=DKIM1; p=${DKIM_PUB_KEY}\"" ] }, { "subname": "mta-sts", "type": "CNAME", "ttl": 3600, "records": [ "mail.${DOMAIN_NAME}." ] } ]</code></pre>Add Data to deSEC🔗</a></h1> There are now two more environment variables necessary before the call to the deSEC API is possible. DKIM_PUB_KEY</code> is the dkim key from your mail system. for example with amavis /usr/sbin/amavisd showkeys ${DOMAIN_NAME}</code></li> </ul> </li> TLSA_HASH</code> is the sha256 hash for the TLSA</code> Record.</li> </ul> example call🔗</a></h2> The shell code below shows how such a shell session looks like. warning </div> Example </div> You will need to adopt the environment variables. </div> </div> export DNSSEC_TOKEN='Your-DNSSEC-API-Token' export DOMAIN_NAME='Your-Domain' export DKIM_PUB_KEY='Your-DKIM-Public-Key' export TLSA_HASH=$(openssl x509 -noout -pubkey \ -in /etc/letsencrypt/live/${DOMAIN_NAME}/fullchain.pem \ | openssl pkey -pubin -outform DER \ | openssl sha256 2>&1 \ | cut -f2 -d ' ') envsubst < mail-setup.json|curl -sSLX PUT \ https://desec.io/api/v1/domains/${DOMAIN_NAME}/rrsets/ \ --header "Authorization: Token ${DNSSEC_TOKEN}" \ --header "Content-Type: application/json" --data @- | jq</code></pre> tls-proxy-tunnel: Transparent TLS Tunnelling Through Corporate HTTP Proxies 2024-07-02T00:00:00+00:00 Many enterprise networks sit behind a corporate HTTP CONNECT proxy. Applications that speak TLS natively — think git</code>, curl</code>, SSH-over-HTTPS, or any custom binary — often have no built-in proxy support. Configuring every single tool is tedious, fragile, and sometimes impossible when you don’t control the binary. tls-proxy-tunnel</code> (tpt</code>) solves this at layer 4: it sits between your application and the outside world, intercepts the TLS connection, extracts the Server Name Indication (SNI) from the ClientHello, and tunnels the raw bytes through your corporate HTTP CONNECT proxy — without ever terminating TLS. Building a native file upload handler for Caddy v2 2022-03-01T00:00:00+00:00 When Caddy v2</a> was released in 2020, it introduced a new concept of handlers that use Go’s native module features. In 2022 one of our customers needed to upload files, and at that time none of the available web servers could handle this natively without an extra program. So I decided to create a new handler for Caddy v2 — caddyv2-upload</a> was born. I was not aware of any other tool which can handle file uploads natively in just one binary which can be used in Kubernetes or OpenShift, or I have not searched hard enough 😁. I also wanted to learn go, so I decided to create a new handler for the go based webserver caddyserver</a>. In May 2022 dufs</a>, a Rust-based server, was created — so that might be another option worth considering. 🤷 Any how I wanted to learn go in the pre AI Area and at that time I looked into the docs of go modules, into the source of caddy servers own handlers and created step by step a new handler. Luckily there was already podman and buildah, so I was able to create container images to run and test locally. Building something real turned out to be the most effective way to learn Go. Diving into Caddy’s own handler source code, understanding the module system, and iterating with containers made the concepts stick far better than any tutorial would have. A minimal Caddyfile configuration looks like this: { order upload before file_server } :2015 { root . file_server browse @mypost method POST upload @mypost { dest_dir /var/uploads max_filesize 4MB } }</code></pre> Build with xcaddy</a>: xcaddy build --with github.com/git001/caddyv2-upload</code></pre> Today caddyv2-upload</a> (v0.20.0) supports file uploads via HTTP multipart form, configurable destination directories, UUID-based subdirectory creation per upload, fixed filenames, pass-through mode to continue to the next Caddy handler after upload, configurable file size limits, an HTTP notification callback after successful uploads, and a /health</code> endpoint. It integrates with Caddy’s variable system and works in Kubernetes and OpenShift environments. Authentication is handled by Caddy’s own security layer rather than the module itself. How does SNI Routing work in HAProxy 2019-05-17T00:00:00+00:00 As I travel a lot I faced the problem that in some Wi-Fi networks certain ports are blocked for outgoing communication 😱. The solution is to use software that can handle TCP and HTTP via port 443 so that I can use my Nextcloud and my XMPP client on the same port.

Flag</th>	ENV</th>	Subcommands</th></tr></thead>
`--deployment</code></td>`	`K8S_SCALE_DEPLOYMENT</code></td>`	both</td></tr>
`--namespace</code></td>`	`K8S_SCALE_NAMESPACE</code></td>`	both</td></tr>
`--replicas</code></td>`	`K8S_SCALE_REPLICAS</code></td>`	scale</td></tr>
`--dry-run</code></td>`	`K8S_SCALE_DRY_RUN</code></td>`	both</td></tr>
`--extra-ca-bundle</code></td>`	`K8S_SCALE_EXTRA_CA_BUNDLE</code></td>`	both</td></tr> </tbody></table> The `--namespace</code> flag exists, but the CronJob does not hardcode it. Instead, the pod reads its own namespace from the Downward API and passes it through as an environment variable. That keeps the ServiceAccount’s Role safely namespace-scoped: the tool can only ever act on Deployments in the namespace where the CronJob itself runs. Not a general-purpose “scale anything anywhere” tool.</p>` Design highlights🔗</a></h2> Minimal RBAC via the scale</code> subresource🔗</a></h3> Setting replicas does not need patch</code> on the full Deployment. Kubernetes exposes deployments/scale</code> as a dedicated subresource:</p> r</span>ules</span>:</span></span> -</span> a</span>piGroups</span>:</span> [</span>"</span>apps</span>"</span>]</span></span> r</span>esources</span>:</span> [</span>"</span>deployments/scale</span>"</span>]</span></span> v</span>erbs</span>:</span> [</span>"</span>get</span>"</span>,</span> "</span>patch</span>"</span>,</span> "</span>update</span>"</span>]</span></span></code></pre> That is all the scale mode needs. Restart mode does need patch</code> on deployments</code> itself (to update the pod template annotation), so the shipped Role covers both:</p> -</span> a</span>piGroups</span>:</span> [</span>"</span>apps</span>"</span>]</span></span> r</span>esources</span>:</span> [</span>"</span>deployments</span>"</span>]</span></span> v</span>erbs</span>:</span> [</span>"</span>get</span>"</span>,</span> "</span>patch</span>"</span>]</span></span></code></pre> patch</code> on the full Deployment is broader than patch</code> on deployments/scale</code> — RBAC review at deploy time should factor that in when the CronJob is configured for restart mode.</p> Extra-CA bundle merged into the trust store🔗</a></h3> The kube client uses the in-cluster CA from the ServiceAccount token secret automatically. For clusters whose API server certificate is signed by a corporate CA that is not in the SA-mounted ca.crt</code>, --extra-ca-bundle <path></code> reads a PEM file (or a full chain of them) and appends every CERTIFICATE</code> block to kube::Config.root_cert</code>. Non-certificate blocks in the file are filtered out.</p> Rendered in Helm as an optional ConfigMap mount; in Kustomize as an opt-in Component. Neither path forces the mount when there is no corporate CA to add.</p> The rustls 0.23 CryptoProvider gotcha🔗</a></h3> If you build against kube-rs</code> with the rustls-tls</code> feature on the current release, the binary compiles cleanly — and then panics at the first TLS handshake with:</p> Could not automatically determine the process-level CryptoProvider from Rustls crate features</span></span></code></pre> rustls</code> 0.23 no longer picks a crypto backend based on downstream feature flags. The application itself must call rustls::crypto::ring::default_provider().install_default()</code> before any TLS work happens. In k8s-scale-app-rs this runs as the first step in main()</code>.</p> Two notes for anyone hitting the same wall: the panic never fires at compile time, and no test caught it locally because integration tests without a cluster do not open a TLS connection. It only surfaces when the CronJob actually tries to reach the API server.</p> Two deploy paths, independent🔗</a></h3> Helm and Kustomize both ship in the repo, and either can be used on its own:</p> Helm chart</strong> with values-{dev,preprod,prod}.yaml</code>, a mode: scale \| restart</code> toggle that swaps the subcommand argument and drops the unused replicas env, and serviceAccount.create</code> / rbac.create</code> / extraCA.enabled</code> toggles for the pieces that a given cluster may or may not need.</li> Kustomize</strong> with a base that has only the CronJob, plus three opt-in Components (serviceaccount/</code>, rbac/</code>, extra-ca/</code>) that overlays include as needed. Overlays under overlays/{dev,preprod,prod}/</code> set namespace and image tag and a patch that touches only the fields that vary per stage.</li> </ul> No Helm-inside-Kustomize glue, no Kustomize-on-Helm-render. Whichever tool matches an existing pipeline is the one to use.</p> Two container images🔗</a></h2> The default image and a -debug</code> variant, both built in the same GitHub Actions matrix — the production/-debug</code> split itself is a general pattern, covered in more depth in Part 2 of the K8s/OpenShift Best Practices series</a>:</p> File</th> Base</th> Contents</th> Approx. size</th></tr></thead> Containerfile</code></td> registry.access.redhat.com/ubi10/ubi-micro</code></td> Binary + CA trust store + /licenses/</code>, no package manager</td> ≈35 MB</td></tr> Containerfile.debug</code></td> registry.access.redhat.com/ubi10/ubi-minimal</code></td> Everything above + bash</code>, curl</code>, bind-utils</code> (dig</code>)</td> ≈115 MB</td></tr> </tbody></table> ubi10-micro</code> is not fully distroless in the Google sense — bash and coreutils are still there — but it has no package manager and no network tools, which fits the pod’s actual operational surface for a scheduled scale or restart call. The -debug</code> variant exists for the day a pod loops in ImagePullBackOff</code> or a network policy silently drops the API-server traffic and someone needs to kubectl exec</code> in with tools to check.</p> Why UBI10🔗</a></h3> The base itself, not just the two variants built from it, was a deliberate choice. I went with Red Hat’s UBI over an Alpine or a generic Docker Hub base image for one reason: it gives the image a level of maintenance stability I don’t have to police myself — CVE scanning and rebuilds are Red Hat’s job, not something I track on my own. That comes with a tradeoff worth stating plainly: UBI is not “digitally sovereign” in the sense the upcoming Digital Sovereignty in Practice series covers — it is a US vendor’s supply chain, the same as pulling from Docker Hub or almost any other public registry would be. I did not find a base that solved that concern without giving up the maintenance guarantees I wanted, so UBI10 here is a compromise I made deliberately, not a claim that the underlying problem is solved.</p> Both images ship an aggregated /licenses/LICENSES.txt</code> bundled with cargo-about</code> during the container build, generated from a whitelist of accepted SPDX identifiers. A new dependency that pulls in an unfamiliar license fails the container build on purpose.</p> Supply chain🔗</a></h2> The build-publish workflow</a> runs three jobs on every push, plus a release job on tag pushes:</p> test</code></strong> — cargo fmt --check</code>, cargo clippy -D warnings</code>, cargo test --release</code>. Cluster tests auto-skip in CI (no KUBECONFIG</code>, no in-cluster SA token).</li> build-image</code></strong> — matrix over Containerfile</code> and Containerfile.debug</code>. Each variant is built with buildx, pushed to ghcr.io/git001/k8s-scale-app-rs</code>, smoke-tested (--version</code>, scale --help</code>, restart --help</code>), then: signed with cosign keyless</strong> via the GitHub OIDC token (no key management — Fulcio issues a short-lived certificate, the signature is logged to Rekor);</li> has an SPDX-JSON SBOM</strong> generated with syft and attached via cosign attest --type spdxjson</code>;</li> has SLSA build provenance</strong> pushed with actions/attest-build-provenance</code>.</li> </ul> </li> publish-chart</code></strong> — helm lint</code> and helm package</code> on every push; OCI push to oci://ghcr.io/git001/charts/k8s-scale-app-rs</code>, plus cosign sign and SLSA provenance only on tag events (avoids overwrite conflicts on GHCR-immutable tags).</li> release</code></strong> (tag events only) — creates the GitHub Release with auto-generated notes from commits since the previous tag, attaches the packaged chart .tgz</code> as an asset, and prepends the release body with pull and verify snippets.</li> </ol> All of that runs with permissions: contents:read, packages:write, id-token:write, attestations:write</code> — the minimum needed for OIDC-based signing plus attestation upload.</p> Verifying a published image🔗</a></h2> A consumer that wants to check what came out of that pipeline needs cosign v2:</p> IMG</span>=</span>g</span>h</span>c</span>r</span>.</span>i</span>o</span>/</span>g</span>i</span>t</span>0</span>0</span>1</span>/</span>k</span>8</span>s</span>-</span>s</span>c</span>a</span>l</span>e</span>-</span>a</span>p</span>p</span>-</span>r</span>s</span>:</span>v</span>1</span>.</span>0</span>.</span>0</span></span> IDENTITY</span>=</span>'</span>^https://github.com/git001/k8s-scale-app-rs/</span>'</span></span> ISSUER</span>=</span>h</span>t</span>t</span>p</span>s</span>:</span>/</span>/</span>t</span>o</span>k</span>e</span>n</span>.</span>a</span>c</span>t</span>i</span>o</span>n</span>s</span>.</span>g</span>i</span>t</span>h</span>u</span>b</span>u</span>s</span>e</span>r</span>c</span>o</span>n</span>t</span>e</span>n</span>t</span>.</span>c</span>o</span>m</span></span> </span> cosign</span> verify</span> \</span></span> -</span>-certificate-identity-regexp</span> "</span>$</span>IDENTITY</span>"</span> \</span></span> -</span>-certificate-oidc-issuer</span> "</span>$</span>ISSUER</span>"</span> \</span></span> "</span>$</span>IMG</span>"</span></span> </span> cosign</span> verify-attestation</span> -</span>-type</span> spdxjson</span> \</span></span> -</span>-certificate-identity-regexp</span> "</span>$</span>IDENTITY</span>"</span> \</span></span> -</span>-certificate-oidc-issuer</span> "</span>$</span>ISSUER</span>"</span> \</span></span> "</span>$</span>IMG</span>"</span></span> </span> cosign</span> verify-attestation</span> -</span>-type</span> slsaprovenance</span> \</span></span> -</span>-certificate-identity-regexp</span> "</span>$</span>IDENTITY</span>"</span> \</span></span> -</span>-certificate-oidc-issuer</span> "</span>$</span>ISSUER</span>"</span> \</span></span> "</span>$</span>IMG</span>"</span></span></code></pre> For enforcement inside the cluster, a cosign-aware admission controller — Kyverno</a>, Sigstore policy-controller</a>, or Connaisseur</a> — can turn the signature into a hard gate that rejects any pod trying to pull an image that was not produced by exactly this workflow. That is a separate deployment concern; the tool repo does not ship the policy itself.</p> Mirroring the image into another registry🔗</a></h2> Verifying a signature only works if the signature actually travels with the image. Not every docker pull && docker push</code>-style mirroring tool preserves it — some tools do, some quietly drop it, and one (oc-mirror v2</code>) solves a related but entirely different signature problem. I wrote up a full, sourced comparison — Which Tool Mirrors a Cosign-Signed Image into a Private Registry?</a> — with regctl</code> coming out as the most complete option for this specific image.</p> Code and repository🔗</a></h2> Repository: k8s-scale-app-rs on GitHub</a></li> Container images: ghcr.io/git001/k8s-scale-app-rs:v1.0.0</code> and ghcr.io/git001/k8s-scale-app-rs:v1.0.0-debug</code></li> Helm chart: oci://ghcr.io/git001/charts/k8s-scale-app-rs:1.0.0</code></li> </ul> Feedback and PRs welcome.</p> Which Tool Mirrors a Cosign-Signed Image into a Private Registry? 2026-07-06T00:00:00+00:00 TL;DR🔗</a></h2> regctl</code> is the most complete option — it exposes the legacy tag-based signature scheme and the modern OCI 1.1 referrers API as two separate, explicit flags. oras cp -r</code> is the second choice, and the tool cosign’s own project points to now that cosign copy</code> is deprecated. oc-mirror v2</code> looks like it should be on this list but actually solves a different, unrelated signature problem.</p> The problem🔗</a></h2> Verifying a signature only works if the signature actually travels with the image. Anyone mirroring a Cosign-signed image into a private or air-gapped registry needs the copy step itself to carry the signature along, not just the layers — and that is not automatic. Some tools drop it silently, some need a config flag turned on first, and one widely-used tool solves a related but entirely different problem. k8s-scale-app-rs</a> is the worked example throughout this post — its own container images are signed keylessly with cosign, carry an SBOM and SLSA build provenance, and its “Verifying a published image”</a> section shows what verification actually checks. This post is about what happens before</em> verification: getting the signature into the registry the verifier will actually query.</p> A short, sourced comparison, checked directly against each project’s docs/source (as of July 2026):</p> A naming trap worth knowing first🔗</a></h2> Red Hat’s oc-mirror v2</code> — the tool for mirroring into disconnected OpenShift installs — has its own --secure-policy</code>/--remove-signatures</code>/--registries.d</code> flags, but they implement Red Hat’s older “simple signing” scheme: a policy.json</code> plus GPG keys plus an external sigstore:</code> URL (e.g. mirror.openshift.com/pub/openshift-v4/signatures</code> for OpenShift release images). That predates, and has nothing to do with, the CNCF Sigstore project this post’s cosign workflow uses — same word, unrelated mechanism. oc-mirror v2</code> went GA in OpenShift 4.18 (release notes</a>), but its ImageSetConfiguration</code> API is still v2alpha1</code>, and none of Red Hat’s own walkthroughs for it touch Cosign-style signatures at all.</p> Tool comparison🔗</a></h2> Tool</th> Legacy tag-based signature (sha256-<digest>.sig</code>)</th> OCI 1.1 referrers (modern Cosign/SBOM/attestations)</th> Note</th></tr></thead> regctl image copy</code></a></td> --digest-tags</code></td> --referrers</code></td> only tool here with separate, explicit flags for both schemes</td></tr> oras cp -r</code></a></td> via -r</code></td> -r</code> (marked “Preview”)</td> --from/to-distribution-spec</code> handles referrers-tag vs. referrers-API mismatches between registries</td></tr> skopeo copy</code></td> opt-in</td> not supported</td> needs use-sigstore-attachments: true</code> under the default-docker</code> key in registries.d</code>, on both source and destination</td></tr> Zot registry sync</code></td> syncLegacyCosignTags</code> (default true</code>)</td> only with preserveDigest: true</code></td> without preserveDigest</code>, Zot re-encodes Docker-format images to OCI on sync, which changes the digest and detaches the signature</td></tr> cosign copy</code></td> yes</td> no</td> deprecated</strong> as of sigstore/cosign#4681</a> (merged 2026-02-04) — the tool itself now points to oras copy -r</code> instead</td></tr> crane copy</code></td> only incidentally, via --all-tags</code></td> no</td> plain manifest/layer copier, no signature awareness at all</td></tr> Harbor replication</td> per Harbor’s own docs, yes</td> open gap (goharbor/harbor#23210</a>)</td> works for the legacy scheme; OCI 1.1 referrers replication is not yet reliable</td></tr> </tbody></table> regctl</code> is the only tool in this comparison that exposes both signature schemes as distinct, intentional flags rather than as a side effect of copying “everything.” For a registry pair that already speaks the OCI 1.1 referrers API end to end, oras cp -r</code> is the other option worth checking — it’s what the cosign project’s own deprecation notice for cosign copy</code> points to.</p> Example: mirroring with regctl🔗</a></h2> Here is an example command to mirror k8s-scale-app-rs</code> itself, signature included, using regctl</code>:</p> regctl</span> image</span> copy</span> \</span></span> -</span>-digest-tags</span> -</span>-referrers</span> \</span></span> ghcr.io/git001/k8s-scale-app-rs:v1.0.0</span> \</span></span> my-private-registry.example.com/k8s-scale-app-rs:v1.0.0</span></span></code></pre>Sources🔗</a></h2> sigstore/cosign PR #4681 — “Deprecate cosign copy”</a></li> sigstore/cosign#4335 — Complete OCI 1.1 Referrers API Support Across All Cosign Commands</a></li> sigstore/cosign#4564 — cosign verify fails after cosign copy (JFrog Artifactory)</a></li> podman-container-tools/skopeo#2061 — sigstore signature copying and use-sigstore-attachments</code></a></li> regctl image copy</code> reference</a></li> oras cp</code> command reference</a></li> goharbor/harbor#23210 — OCI 1.1 referrers not replicated</a></li> Zot registry: mirroring/sync documentation</a></li> OpenShift Container Platform 4.18 release notes</a></li> openshift/oc-mirror — signature-verification.md</a></li> </ul> haproxy-spoe-rs: Deployment 2026-04-12T00:00:00+00:00 This post covers deploying the haproxy-spoe-rs</a> SPOA agent in production. It is a companion to the architecture post</a> which explains the library design.</p> The agent and HAProxy run as separate services</strong> — HAProxy connects to the agent over TCP. This keeps their lifecycles independent and allows scaling each side separately.</p> Running pdns_recursor as a root-independent validating resolver 2026-04-06T00:00:00+00:00 PowerDNS Recursor</a> is a mature, production-grade recursive DNS resolver. This post documents how to configure it to bootstrap directly from a local root.zone</code> file — so it never needs to reach the root name servers at runtime — with full DNSSEC validation producing the ad</code> flag in responses.</p> Running a validating DNS recursor from the root zone with Hickory DNS 2026-04-04T00:00:00+00:00 Hickory DNS</a> is a Rust-based DNS implementation that covers the full stack: resolver, recursor, authoritative server, and DNSSEC. This post documents how to run it as a full recursive resolver</strong> — starting directly from the root zone, with DNSSEC validation and encrypted upstream transport — based on the changes developed in the recurser-from-root-zone</code></a> branch.</p> vigil-log-relay: Kubernetes Log Collection Without a DaemonSet 2026-03-22T00:00:00+00:00 Every team running workloads on Kubernetes eventually faces the same question: how do you get logs out of your pods and into your log aggregation stack? The standard answer is a DaemonSet. It works, but it comes with a cost. vigil-log-relay</a></strong> is a different answer.</p> vigil-rs: A Rust Service Supervisor for Containers 2026-03-21T00:00:00+00:00 Every container that runs more than one process needs a supervisor. Most reach for dumb-init</code>, tini</code>, s6-overlay</code>, or supervisord</code>. None of them felt quite right for what we needed, so we wrote vigil-rs</strong>.</p> This post walks through what github.com/git001/vigil-rs</a> is, why we built it, and how it works internally.</p> mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster 2026-03-15T00:00:00+00:00 mergelog</a> is a small but handy tool: it takes multiple Apache log files from servers behind a round-robin DNS and merges them into a single chronologically sorted stream. The original version 4.5 was written in C around 2000–2001, weighs in at ~450 lines, and does exactly what it promises — reliably and fast.</p> This post documents the journey from mergelog-4.5 to mergelog-rs</a>: what changed, what was surprising, and how six targeted optimizations made the Rust binary 2.26× faster</strong> than the C original.</p> Full respect and honour to Bertrand Demiddelaer</strong>, who created mergelog back in 2000. Writing a tool this fast, this correct, and this compact in C — with no external dependencies beyond zlib — is genuinely impressive. mergelog-rs stands on his shoulders.</p> The Original: mergelog 4.5🔗</a></h2> The C program reads any number of log files in Common Log Format</em> (CLF) / NCSA Combined Format</em>:</p> %h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i"</span></span></code></pre> Example line:</p> 93.184.216.34 - - [15/Mar/2026:12:00:00 +0100] "GET /index.html HTTP/1.1" 200 1234 "-" "curl/7.88"</span></span></code></pre> The algorithm in 4.5 is straightforward: find the earliest timestamp across all files, then iterate second-by-second from there to the end, writing all matching lines at each step. That is O(timespan × K)</strong> — slow when logs span multiple years or many files are involved.</p> Gzip support exists, but as a separate binary (zmergelog</code>) compiled with -DUSE_ZLIB</code>.</p> The Rust Rewrite: What Changed🔗</a></h2> 1. Algorithm: K-Way Merge with a Binary Heap🔗</a></h3> Instead of time-based iteration, mergelog-rs uses a k-way merge with a min-heap</strong> (std::collections::BinaryHeap</code> with reversed ordering):</p> Seed the heap with the first line from each file</li> Main loop: pop the smallest timestamp, write the line, push the next line from the same file</li> Complexity: O(N log K)</strong> — independent of the time span of the logs</li> </ul> while</span> let</span> Some</span>(</span>Entry</span> {</span> mut</span> line</span>,</span> mut</span> reader</span>,</span> file_idx</span>,</span> ..</span> }</span>)</span> =</span> heap</span>.</span>pop</span>(</span>)</span> {</span></span> out</span>.</span>write_all</span>(</span>line</span>.</span>as_bytes</span>(</span>)</span>)</span>?</span>;</span></span> line</span>.</span>clear</span>(</span>)</span>;</span></span> if</span> fill_line</span>(</span>&</span>mut</span> </span>reader</span>,</span> &</span>mut</span> line</span>)</span>?</span> {</span></span> let</span> ts</span> =</span> parse_clf_timestamp</span>(</span>&</span>line</span>)</span>.</span>unwrap_or_else</span>(</span>sentinel</span>)</span>;</span></span> heap</span>.</span>push</span>(</span>Entry</span> {</span> ts</span>,</span> file_idx</span>,</span> line</span>,</span> reader</span> }</span>)</span>;</span></span> }</span></span> }</span></span></code></pre>2. Magic-Byte Detection Instead of File Extension🔗</a></h3> The original detects gzip by binary type (zmergelog</code> vs. mergelog</code>). mergelog-rs reads the first 6 bytes of each file and detects the format automatically:</p> Format</th> Magic Bytes</th></tr></thead> gzip</td> 1F 8B</code></td></tr> bzip2</td> 42 5A 68</code> (BZh</code>)</td></tr> xz</td> FD 37 7A 58 5A 00</code></td></tr> zstd</td> 28 B5 2F FD</code></td></tr> </tbody></table> pub</span> fn</span> detect_from_bytes</span>(</span>magic</span>:</span> &</span>[</span>u8</span>]</span>)</span> -></span> Compression</span> {</span></span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 2</span> &&</span> magic</span>[</span>..</span>2</span>]</span> ==</span> [</span>0x1f</span>,</span> 0x8b</span>]</span> {</span> Compression</span>::</span>Gz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 3</span> &&</span> magic</span>[</span>..</span>3</span>]</span> ==</span> [</span>0x42</span>,</span> 0x5a</span>,</span> 0x68</span>]</span> {</span> Compression</span>::</span>Bz2</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 6</span> &&</span> magic</span>[</span>..</span>6</span>]</span> ==</span> [</span>0xfd</span>,</span> 0x37</span>,</span> 0x7a</span>,</span> 0x58</span>,</span> 0x5a</span>,</span> 0x00</span>]</span> {</span> Compression</span>::</span>Xz</span> }</span></span> else</span> if</span> magic</span>.</span>len</span>(</span>)</span> >=</span> 4</span> &&</span> magic</span>[</span>..</span>4</span>]</span> ==</span> [</span>0x28</span>,</span> 0xb5</span>,</span> 0x2f</span>,</span> 0xfd</span>]</span> {</span> Compression</span>::</span>Zstd</span> }</span></span> else</span> {</span> Compression</span>::</span>None</span> }</span></span> }</span></span></code></pre> One binary, all formats, no file extension required. The same peek-based detection also works for stdin — BufReader::fill_buf()</code> inspects the first bytes without consuming them, so no seek is needed.</p> 3. Timezone-Aware Timestamp Comparison🔗</a></h3> The C original ignores the timezone offset in timestamps — it converts date components to local time via mktime()</code>. For logs from servers in different timezones this can produce incorrectly ordered output.</p> mergelog-rs converts every timestamp to UTC before comparison. A +0900</code> timestamp and a -0500</code> timestamp are ordered correctly, regardless of the system’s local timezone.</p> Profiling: Where Does the Time Go?🔗</a></h2> Since perf</code> was locked down on the system (perf_event_paranoid = 4</code>), a dedicated timing binary was built that measures each phase manually with std::time::Instant</code>:</p> ──── profile_timing (41.65M lines) ──────────────────</span></span> I/O (read_line) 3.91 s 32.4%</span></span> Parse (jiff strptime) 5.52 s 45.7% ← bottleneck</span></span> Heap (push/pop) 1.97 s 16.3%</span></span> Write (write_all) 0.69 s 5.7%</span></span> ─────────────────────────────────────────────────</span></span> Σ 12.08 s</span></span></code></pre> Nearly half the time was in the parser.</strong> The reason: jiff::Zoned::strptime</code> is a full parsing pipeline — format-string interpretation, timezone database lookup, error handling, heap allocation. Across ~42 million lines that adds up fast.</p> This full parsing pipeline is super handy, but in our use case it is not necessary — therefore we built a hand-rolled CLF timestamp parser.</p> Optimization 1: Hand-Rolled CLF Parser🔗</a></h2> The CLF timestamp format is completely fixed:</p> [DD/Mon/YYYY:HH:MM:SS ±HHMM]</span></span> 0123456789012345678901234 5</span></span></code></pre> A hand-written parser only needs to:</p> Find [</code> in the string</li> Read 6 integers from fixed byte positions</li> Compute the Unix timestamp via integer arithmetic</li> </ol> For step 3 we use Howard Hinnant’s proleptic Gregorian algorithm — no branches, pure multiplication:</p> fn</span> days_since_epoch</span>(</span>y</span>:</span> i64</span>,</span> m</span>:</span> i64</span>,</span> d</span>:</span> i64</span>)</span> -></span> i64</span> {</span></span> let</span> y</span> =</span> if</span> m</span> <=</span> 2</span> {</span> y</span> -</span> 1</span> }</span> else</span> {</span> y</span> }</span>;</span></span> let</span> era</span> =</span> if</span> y</span> >=</span> 0</span> {</span> y</span> }</span> else</span> {</span> y</span> -</span> 399</span> }</span> /</span> 400</span>;</span></span> let</span> yoe</span> =</span> y</span> -</span> era</span> </span> 400</span>;</span></span> let</span> doy</span> =</span> (</span>153</span> </span> (</span>m</span> +</span> if</span> m</span> ></span> 2</span> {</span> -</span>3</span> }</span> else</span> {</span> 9</span> }</span>)</span> +</span> 2</span>)</span> /</span> 5</span> +</span> d</span> -</span> 1</span>;</span></span> let</span> doe</span> =</span> yoe</span> </span> 365</span> +</span> yoe</span> /</span> 4</span> -</span> yoe</span> /</span> 100</span> +</span> doy</span>;</span></span> era</span> </span> 146_097</span> +</span> doe</span> -</span> 719_468</span></span> }</span></span></code></pre> Result:</strong> parse time dropped from 5.52 s to 1.50 s — 3.7× faster</strong>.</p> Optimization 2: Reuse the String Buffer🔗</a></h2> The original implementation allocated a new String</code> for every line read:</p> //</span> before: fresh allocation per line</span></span> fn</span> read_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>)</span> -></span> Result</span><</span>Option</span><</span>String</span>></span>></span> {</span></span> let</span> mut</span> line</span> =</span> String</span>::</span>new</span>(</span>)</span>;</span> //</span> ← new every time</span></span> reader</span>.</span>read_line</span>(</span>&</span>mut</span> line</span>)</span>?</span>;</span></span> ...</span></span> }</span></span></code></pre> The fix is simple: when an Entry</code> is popped from the heap, you receive line: String</code> with full ownership. Instead of dropping it, clear it and read the next line directly into the same buffer:</p> //</span> after: capacity is retained</span></span> while</span> let</span> Some</span>(</span>Entry</span> {</span> mut</span> line</span>,</span> mut</span> reader</span>,</span> file_idx</span>,</span> ..</span> }</span>)</span> =</span> heap</span>.</span>pop</span>(</span>)</span> {</span></span> out</span>.</span>write_all</span>(</span>line</span>.</span>as_bytes</span>(</span>)</span>)</span>?</span>;</span></span> line</span>.</span>clear</span>(</span>)</span>;</span> //</span> ← keeps the heap allocation</span></span> if</span> fill_line</span>(</span>&</span>mut</span> </span>reader</span>,</span> &</span>mut</span> line</span>)</span>?</span> {</span></span> let</span> ts</span> =</span> parse_clf_timestamp</span>(</span>&</span>line</span>)</span>.</span>unwrap_or_else</span>(</span>sentinel</span>)</span>;</span></span> heap</span>.</span>push</span>(</span>Entry</span> {</span> ts</span>,</span> file_idx</span>,</span> line</span>,</span> reader</span> }</span>)</span>;</span></span> }</span></span> }</span></span></code></pre> Since consecutive log lines are similar in length, the buffer stops growing after the first few lines — no malloc</code> calls in the hot path.</p> Optimization 3: mimalloc as the Global Allocator🔗</a></h2> With most per-line allocations already eliminated, there are still smaller allocations happening — buffer growth during seeding, internal BinaryHeap</code> resizing, and the occasional String</code> capacity bump. Swapping out the default glibc allocator for mimalloc</a> is a one-liner in Rust:</p> #</span> Cargo.toml</span></span> mimalloc</span> =</span> {</span> version</span> =</span> "</span>0.1</span>"</span>,</span> default-features</span> =</span> false</span> }</span></span></code></pre>//</span> main.rs</span></span> use</span> mimalloc</span>::</span>MiMalloc</span>;</span></span> #</span>[</span>global_allocator</span>]</span></span> static</span> GLOBAL</span>:</span> MiMalloc</span> =</span> MiMalloc</span>;</span></span></code></pre> That’s it — no other code changes. mimalloc is designed for throughput-heavy workloads with many small, short-lived allocations, which matches our remaining allocation pattern well.</p> Result:</strong> 5.55 s → 5.39 s — a further ~3% improvement.</p> Not a dramatic win on its own, but it’s essentially free: two lines of code, no logic changes, and it stacks on top of everything else.</p> Optimization 4: 4 MiB Read Buffers + SIMD Newline Search🔗</a></h2> After the previous three optimizations, profiling showed I/O was still the largest remaining cost at 47% of total time. Two changes address this together.</p> 4 MiB read buffers</strong> — The previous BufReader</code> used a 256 KiB internal buffer, meaning a syscall every ~256 KiB of data. Increasing to 4 MiB reduces syscall frequency by 16×, which shows up directly in system time.</p> SIMD newline search via memchr</code></strong> — Instead of BufReader::read_line</code>, the line reader now uses fill_buf()</code> + memchr::memchr()</code> directly. The memchr</code> crate uses AVX2/SSE2 to scan 16 or 32 bytes per cycle when searching for \n</code>:</p> fn</span> fill_line</span>(</span>reader</span>:</span> &</span>mut</span> dyn</span> BufRead</span>,</span> buf</span>:</span> &</span>mut</span> String</span>)</span> -></span> Result</span><</span>bool</span>></span> {</span></span> loop</span> {</span></span> let</span> available</span> =</span> reader</span>.</span>fill_buf</span>(</span>)</span>?</span>;</span></span> if</span> available</span>.</span>is_empty</span>(</span>)</span> {</span> return</span> Ok</span>(</span>false</span>)</span>;</span> }</span></span> match</span> memchr</span>(</span>b</span>'</span>\</span>n</span>'</span>,</span> available</span>)</span> {</span></span> Some</span>(</span>pos</span>)</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>&</span>available</span>[</span>..</span>pos</span> +</span> 1</span>]</span>)</span>?</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>pos</span> +</span> 1</span>)</span>;</span></span> return</span> Ok</span>(</span>true</span>)</span>;</span></span> }</span></span> None</span> =></span> {</span></span> buf</span>.</span>push_str</span>(</span>std</span>::</span>str</span>::</span>from_utf8</span>(</span>available</span>)</span>?</span>)</span>;</span></span> let</span> len</span> =</span> available</span>.</span>len</span>(</span>)</span>;</span></span> reader</span>.</span>consume</span>(</span>len</span>)</span>;</span></span> }</span></span> }</span></span> }</span></span> }</span></span></code></pre> Result:</strong> 5.39 s → 4.43 s — another 18% improvement</strong>, bringing the total to 2.26× faster</strong> than the C original.</p> A Road Not Taken: Parallel Reader Threads🔗</a></h2> An obvious next step seemed to be spawning one thread per input file — each thread handles its own I/O and decompression, sending (timestamp, line)</code> pairs to the merge thread via a bounded mpsc</code> channel:</p> Thread 1: read + decompress + parse → channel ─┐</span></span> Thread 2: read + decompress + parse → channel ─┤→ Merge thread → stdout</span></span> ... ─┘</span></span></code></pre> For compressed files</strong> (bz2, xz) this is conceptually attractive: decompression is CPU-bound and can genuinely run in parallel.</p> For plain-text files</strong> on a single disk, it does not help — the bottleneck is disk read bandwidth, and parallel reads from the same physical device cause seek contention or simply saturate the same I/O bus. Benchmarking confirmed this: the threaded version achieved identical wall-clock time on plain-text files, while adding per-line heap allocation overhead (the channel requires sending owned String</code> values, making buffer reuse impossible across the thread boundary). The result was slightly slower</em> than the sequential version with large buffers.</p> Takeaway:</strong> threading is the right tool when the bottleneck is CPU-bound work that can be parallelised per-file (compressed files on a multi-core system). It is the wrong tool when the bottleneck is a shared I/O resource. Measure first.</p> Final Results🔗</a></h2> Measured on 7 log files × 1 GiB = 7 GiB total, 41.65 million lines:</p> Version</th> Wall time</th> User</th> System</th></tr></thead> mergelog-4.5 (C)</td> 10.03 s</td> 5.93 s</td> 4.12 s</td></tr> mergelog-rs v1 (jiff)</td> 10.39 s</td> 9.13 s</td> 1.25 s</td></tr> + hand-rolled parser</td> 5.99 s</td> 4.75 s</td> 1.23 s</td></tr> + buffer reuse</td> 5.55 s</td> 4.29 s</td> 1.26 s</td></tr> + mimalloc</td> 5.39 s</td> 4.17 s</td> 1.22 s</td></tr> + 4 MiB buffers + memchr</strong></td> 4.43 s</strong></td> 3.61 s</strong></td> 0.81 s</strong></td></tr> </tbody></table> mergelog-rs ran 2.26 ± 0.02 times faster than mergelog-4.5</span></span></code></pre> Memory and CPU Usage🔗</a></h2> Measured with /usr/bin/time -v</code> on the same 7 × 1 GiB workload:</p> </th> mergelog-4.5</th> mergelog-rs</th></tr></thead> Wall time</td> 9.63 s</td> 4.28 s</td></tr> User time</td> 5.61 s</td> 3.42 s</td></tr> System time</td> 4.01 s</td> 0.84 s</td></tr> CPU utilisation</td> 99%</td> 99%</td></tr> Peak RSS</strong></td> 1.6 MiB</strong></td> 30.7 MiB</strong></td></tr> Involuntary ctx switches</td> 105</td> 122</td></tr> </tbody></table> Memory</strong> — The picture changed significantly with the 4 MiB read buffers. mergelog-4.5 uses 1.6 MiB; mergelog-rs now uses 30.7 MiB. The breakdown is straightforward: 7 files × 4 MiB read buffer = 28 MiB, plus the write BufWriter</code>, the binary itself, and mimalloc metadata. This is a deliberate trade-off — RAM is cheap, syscalls are not. At 30 MiB to process 7 GiB of data the ratio is still extremely lean, and the buffer size is a single constant in reader.rs</code> that can be tuned if memory is constrained.</p> System time</strong> — The most striking difference is kernel time: mergelog-4.5 spends 4.01 s in the kernel versus 0.84 s for mergelog-rs. Two factors: the C version calls write(1, ...)</code> directly for every line (~42 million syscalls), and its read buffers are only 32 KiB. mergelog-rs batches writes via BufWriter</code> and reads via 4 MiB BufReader</code>, cutting kernel round-trips dramatically.</p> CPU</strong> — Both tools are single-threaded and max out one core at 99%.</p> Takeaways🔗</a></h2> Five lessons from this project:</p> 1. Measure before optimizing.</strong> Without the timing binary the focus might have landed on the heap. But jiff::strptime</code> was costing nearly 3× more than every other phase combined.</p> 2. Libraries are great until they aren’t.</strong> jiff</code> is an excellent crate for correct timezone-aware date handling. But when you parse the same fixed-format string 42 million times, you pay a steep price for generality. A 30-line hand-rolled parser that exploits the known fixed layout beats it by 3.7×.</p> 3. Sometimes the allocator is the last few percent.</strong> After eliminating almost all allocations from the hot path, swapping in mimalloc took two lines of code and gave another 3% for free. It won’t rescue a poorly written program, but on an already optimized one it’s a cheap final step worth trying.</p> 4. Buffer size matters more than you think.</strong> Going from 256 KiB to 4 MiB read buffers — a one-line change — cut system time by 34% and wall time by 18%. Every syscall is a round-trip to the kernel; fewer is better.</p> 5. Parallelism requires the right bottleneck.</strong> Spawning reader threads per file is conceptually appealing for compressed logs where decompression is CPU-bound. But for plain-text files on a single disk, the bottleneck is I/O bandwidth — adding threads just adds channel overhead and per-line heap allocation, making things slightly slower</em>. Threading is a tool, not a universal solution.</p> 6. Ownership is not a burden.</strong> Buffer reuse in C requires careful explicit lifetime management and is easy to get wrong. In Rust, the ownership system makes it the obvious and safe solution — line.clear()</code> instead of free(line); line = malloc(...)</code>, with the compiler guaranteeing no use-after-free.</p> The code is available under GPL-3.0-or-later. The original mergelog 4.5 was released as GPL-2.0-or-later, which explicitly permits upgrading to any later version of the GPL. GPL-3.0 adds patent protection clauses and anti-tivoization provisions that GPL-2.0 lacks — a straightforward upgrade with no downsides for a command-line tool like this.</p> AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser 2026-03-12T00:00:00+00:00 AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser🔗</a></h1> Background🔗</a></h2> AWFFull 4.0.0</a> is a web server log analyser. It reads Apache/nginx/Squid access logs and produces HTML reports with graphs showing traffic, top URLs, referrers, user agents, country statistics, and more. It was originally forked from The Webalizer</a> by Bradford L. Barrett (1997–2001), extended as AWFFull by Steve McInerney from 2004, and actively developed until 2008, when it was last released as version 3.10.2.</p> It still works well in 2026 — the core log-parsing logic is solid and the output is clean. But the codebase had accumulated 18 years of code: EOL libraries, dead links, obsolete command-line flags, a broken GeoIP implementation, and a very old config.guess</code>. Time for a proper cleanup.</p> This post covers what changed in AWFFull 4.0.0 and why.</p> Is a Static Web Log Analyser Still Useful?🔗</a></h2> Good question. The short answer is: yes — and arguably more so than before.</p> Modern web analytics platforms — Google Analytics, Matomo, Plausible, Grafana with Loki, ELK stacks — are excellent for live dashboards and drill-down exploration. But they share a common weakness: they are slow when you need to process large volumes of historical log data and render the results as a complete set of graphs and tables</strong>.</p> Feed a year of Apache logs (say, 500 million lines across 50 GB of gzipped files) into an ELK pipeline and you’ll be waiting a long time for the data to be ingested, indexed, and ready to query. Then wait again each time you open a dashboard. Grafana rendering a year’s worth of nginx traffic across 30 panels? Expect seconds of load time per page, per query.</p> AWFFull’s approach is different: it processes the raw log files directly, keeps all state in a compact binary history file, and writes static HTML + PNG output</strong> that loads instantly in any browser — even on a machine from 2010, over a slow connection, or offline entirely. There is no query layer, no index, no database, no running server. The report is just files.</p> For the typical use case — a sysadmin or developer who wants a monthly overview of who accessed their server, from where, with what agents, hitting which URLs — this is exactly the right trade-off:</p> Fast to generate</strong>: processing a month of logs takes seconds, not minutes</li> Instant to view</strong>: static HTML, no round-trips, no query engine</li> Self-contained</strong>: the report directory can be archived, moved, or served from any static file server</li> Privacy-friendly</strong>: no third-party JavaScript, no tracking pixels, no data leaving your infrastructure — the raw IP addresses stay in your own logs</li> Works on air-gapped systems</strong>: nothing requires internet access at runtime</li> </ul> It also complements rather than replaces live monitoring. You would typically use Prometheus/Grafana (or similar) to watch your infrastructure in real time, and AWFFull to produce the monthly historical report that goes into a ticket or a management summary.</p> Shipping Access Logs to Object Storage and Running in Containers🔗</a></h3> The operational details — how to ship access logs to S3 or Azure Blob with Fluent Bit, Vector, or Filebeat/Logstash, how to handle concurrent writes and log loss at high traffic, and how to run AWFFull as a stateless container against object storage — are covered in the companion post:</p> AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers</a></strong></p> The Big Ticket Items🔗</a></h2> PCRE → PCRE2🔗</a></h3> AWFFull used libpcre (PCRE version 1) for log-line regex parsing. PCRE 1 has been officially end-of-life since 2012 and is being dropped by distributions. We migrated to PCRE2</strong>, the current maintained successor with a cleaner API and better Unicode support. The change touched parser.c</code>, common.h</code>, and configure.ac</code>.</p> GeoIP: From Legacy .dat</code> to libmaxminddb🔗</a></h3> The old code used libGeoIP</strong> with a proprietary binary .dat</code> format that MaxMind deprecated in 2019. Country lookups were broken for anyone who hadn’t kept a seven-year-old database around.</p> AWFFull 4.0.0 replaces this with libmaxminddb</strong>, the open standard MMDB format used by both MaxMind (GeoLite2) and DB-IP. The migration is transparent — IPv4 and IPv6 are both handled by a single MMDB_lookup_string()</code> call, with no separate code paths needed.</p> Free databases updated monthly:</p> Country</strong>: DB-IP Country Lite</a></li> ASN</strong>: DB-IP ASN Lite</a></li> </ul> ASN Statistics: New Table and Pie Chart🔗</a></h3> Once you have a MMDB-capable lookup layer, adding ASN (Autonomous System Number) statistics is almost free. A second database (GeoASNDatabase</code>) maps IPs to their network operator — think “Cloudflare”, “Amazon AWS”, “Deutsche Telekom”. The report now shows a Top ASNs</strong> table and pie chart, answering the question “which networks send the most traffic to my site?”</em></p> GeoASN yes</span></span> GeoASNDatabase /usr/share/GeoIP/dbip-asn-lite.mmdb</span></span></code></pre> Performance note: both GeoIP and GeoASN lookups happen in the output phase</em>, once per unique IP address — not once per log line. On a typical site the overhead is negligible.</p> Hyperscan: SIMD Pattern Matching🔗</a></h3> AWFFull supports extensive pattern lists for ignoring, grouping, and hiding hosts, URLs, referrers, and agents. The original code used a Boyer-Moore-Horspool</strong> linear scan: each log record was matched against every pattern in turn.</p> When Intel Hyperscan</strong> (libhyperscan</code>) is installed, AWFFull now compiles all patterns from a given list into a single Hyperscan database and matches them in one SIMD pass per record. For large pattern lists — say, 50+ bot/spider agent patterns — this is significantly faster. Hyperscan is auto-detected at configure time and falls back to the existing BMH path if unavailable.</p> sudo</span> apt</span> install</span> libhyperscan-dev</span></span> ./configure</span> &&</span> make</span></span></code></pre>mimalloc: Better Heap Performance🔗</a></h3> For large log files with millions of unique hosts, URLs, and referrers, heap allocation overhead becomes measurable. Adding mimalloc</strong> (Microsoft’s high-performance allocator) as an optional drop-in was straightforward via the explicit mi_calloc</code> API. Auto-detected at configure time.</p> XXH3 Hash Function🔗</a></h3> The internal hash table used djb2, a good general-purpose hash from 1991. AWFFull 4.0.0 replaces it with XXH3</strong> (via the bundled xxhash.h</code>), which offers better distribution and significantly higher throughput on modern CPUs. The table load-factor was also tuned to reduce collisions.</p> User-Visible Improvements🔗</a></h2> Date Stamp on Graphs🔗</a></h3> Every generated PNG graph now shows the current date</strong> (YYYY-MM-DD</code>) in the top-right corner, aligned with the title baseline. Small but useful when you’re looking at archived reports six months later.</p> Next / Previous Month Navigation🔗</a></h3> Monthly report pages now show navigation links at the top:</p> ← February 2026 \| [Index] \| April 2026 →</span></span></code></pre> Links are derived from the history file — only months that actually exist in the database are linked.</p> Updated Browser/Bot Detection🔗</a></h3> The sample.conf</code> browser/agent detection list dated from 2005 and contained entries like “Netscape 4.5”, “Internet Explorer 3.0 (Win95)”, and “MSNBot” — none of which have been seen in the wild for a decade or more. The list has been completely rewritten:</p> Removed:</strong> Netscape 1–4.x, Internet Explorer 3–7, Firebird, Galeon, Camino, msnbot (all entries)</p> Added:</strong> Chrome/Chromium, Edge (Chromium), Brave, Samsung Browser, Firefox, Safari, Opera (OPR/), modern search bots (Bingbot, DuckDuckBot, YandexBot, Baidu, Applebot, AhrefsBot, SemrushBot), AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Amazonbot), and CLI tools (curl, Wget, python-requests).</p> Build System and Tooling🔗</a></h2> CLI Cleanup: 14 Flags Removed🔗</a></h3> AWFFull inherited many single-letter flags from the original Webalizer. Flags like -G</code> (suppress hourly graph), -L</code> (suppress legend), -M</code> (mangle agents), -x</code> (HTML extension), and -I</code> (index alias) had exact equivalents in the config file. At version 4.0.0 — a major version bump — these were removed from the CLI, along with three fully deprecated no-op flags (-d</code>, -q</code>, -Q</code>).</p> The --help</code> output now documents the new verbosity levels (-v</code> through -vvvvv</code>) and includes a note pointing config-file-only settings to awffull.conf(5)</code>.</p> Man Pages: DocBook XML → Pandoc Markdown🔗</a></h3> The existing man pages were written in DocBook XML 4.5 and built with xsltproc</code> — a tool that requires xsltproc</code>, docbook-xsl</code>, and several XML schema packages, none of which are installed by default on most systems in 2026.</p> The man pages are now maintained as pandoc Markdown</strong> (doc/awffull.1.md</code>, doc/awffull.conf.5.md</code>) and built with:</p> pandoc</span> -</span>s</span> -</span>t</span> man</span> awffull.1.md</span> -</span>o</span> awffull.1</span></span></code></pre> configure.ac</code> auto-detects pandoc. Pre-built man pages remain in the repository as a fallback for systems without pandoc. The DocBook XML sources are retained in doc/</code> for historical reference.</p> Translations: Quality Over Quantity🔗</a></h3> AWFFull shipped with 33 language files, most of which were approximately 29% complete and had not been updated since 2008. Rather than shipping incomplete translations that silently fall back to English mid-report, we reduced to 6 fully complete languages</strong> (German, Finnish, Italian, Swedish, Brazilian Portuguese, Indonesian) and brought all of them to 100% (524/524 strings)</strong>.</p> The .pot</code> template was regenerated with xgettext</code> to include all new strings added in 4.0.0 (GeoASN messages, MaxMind DB errors, pattern matching warnings).</p> Repository Modernisation🔗</a></h2> The project moved from a collection of legacy files to a cleaner structure:</p> Before</th> After</th></tr></thead> README</code> + README.FIRST</code> + README.webalizer</code></td> README.md</code> (single entry point)</td></tr> ChangeLog</code> (GNU format, frozen)</td> CHANGELOG.md</code> (Keep a Changelog)</td></tr> COPYING</code></td> LICENSE</code></td></tr> TODO</code> (plain text, 2008)</td> TODO.md</code> (Markdown with [x]</code> checkboxes)</td></tr> country-codes.txt</code> (2005, informational)</td> removed</td></tr> DNS.README</code> (dead link)</td> removed</td></tr> contrib/</code> (Webalizer migration scripts)</td> removed</td></tr> config.guess</code> from 2008</td> updated to automake 1.16 (2022)</td></tr> </tbody></table> tip </div> tip</strong></span> </div> The full list of technical changes is available in CHANGES.md</a>.</p> </div> </div> CI/CD Pipeline🔗</a></h2> A full GitLab CI pipeline was added covering:</p> Build verification</strong> on Debian 12/13, Ubuntu 22.04/24.04, AlmaLinux 8/9/10</li> Source and binary tarballs</strong> on every push to main</code></li> .deb</code> package</strong> (Ubuntu 24.04 amd64) on release tags</li> .rpm</code> package</strong> (AlmaLinux 9 amd64) on release tags</li> Container image</strong> pushed to GitLab Container Registry (Ubuntu 24.04 base, all optional libs included)</li> GitLab Release</strong> creation with all artifacts uploaded to the Generic Package Registry</li> </ul> Releases are triggered by pushing a semver tag:</p> git</span> tag</span> v4.0.0</span></span> git</span> push</span> -</span>-tags</span></span></code></pre> The container image is available as:</p> docker</span> pull</span> registry.gitlab.com/aleks001/awffull:v4.0.0</span></span> docker</span> pull</span> registry.gitlab.com/aleks001/awffull:latest</span></span></code></pre> The other artifacts are available on the Release Page</a></p> What’s Next🔗</a></h2> The TODO.md</code> tracks longer-term ideas: SVG graphs (replacing the aging libgd dependency), weekly summaries, a templating system for the HTML output, and better browser/OS reporting. Contributions welcome.</p> The project is at https://gitlab.com/aleks001/awfull</a>.</p> AWFFull is free software licensed under GPL v3 or later.</em> Based on The Webalizer by Bradford L. Barrett (1997–2001).</em> Extended by Steve McInerney (2004–2008) and Aleksandar Lazic (2026–present).</em></p> AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers 2026-03-12T00:00:00+00:00 This is a companion post to AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser</a>. It covers the operational side: how to get web server access logs into S3 or Azure Blob Storage reliably, and how to run AWFFull inside a container to generate reports from those logs.</p> Shipping Access Logs to Object Storage🔗</a></h2> Before AWFFull can process logs from the cloud, something needs to put them there. The most common approach is to run a log shipper alongside your web server that tails the access log and uploads it to S3 or Azure Blob in date-partitioned batches.</p> </p> danger </div> Critical requirement for AWFFull compatibility</strong></span> </div> The shipper must preserve the raw log lines unchanged — AWFFull needs to see the original Combined/CLF format. Do not parse, transform, or convert the lines to JSON.</p> </div> </div> Fluent Bit → S3🔗</a></h3> Fluent Bit</a> is lightweight, written in C, and has native S3 and Azure Blob output plugins.</p> #</span> /etc/fluent-bit/fluent-bit.conf</span></span> </span> [</span>INPUT</span>]</span></span> Name tail</span></span> Path /var/log/nginx/access.log</span></span> Tag nginx.access</span></span> Read_from_Head false</span></span> Refresh_Interval 5</span></span> </span> [</span>OUTPUT</span>]</span></span> Name s3</span></span> Match nginx.access</span></span> bucket my-logs</span></span> region eu-central-1</span></span> #</span> Date-partitioned path — one directory per day.</span></span> #</span> $UUID is a Fluent Bit built-in variable; it ensures unique filenames</span></span> #</span> when multiple instances run concurrently (e.g. multiple Pods).</span></span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> compression gzip</span></span> total_file_size 50M </span>#</span> flush when file reaches 50 MB</span></span> upload_timeout 10m </span>#</span> or after 10 minutes, whichever comes first</span></span> use_put_object On</span></span></code></pre> For Azure Blob</strong> replace the output section:</p> [</span>OUTPUT</span>]</span></span> Name azure_blob</span></span> Match nginx.access</span></span> account_name mystorageaccount</span></span> shared_key ${AZURE_STORAGE_KEY}</span></span> container_name nginx-logs</span></span> #</span> The azure_blob plugin does not support $UUID in the path parameter.</span></span> #</span> Uniqueness is guaranteed because Fluent Bit appends an internal</span></span> #</span> timestamp + sequence counter to each blob name automatically.</span></span> path nginx/%Y/%m/%d</span></span> auto_create_container On</span></span> tls On</span></span> compress gzip</span></span></code></pre> Vector → S3 / Azure Blob🔗</a></h3> Vector</a> offers more flexibility for transformations and supports both sinks natively:</p> #</span> vector.yaml</span></span> </span> s</span>ources</span>:</span></span> n</span>ginx_access</span>:</span></span> t</span>ype</span>:</span> f</span>ile</span></span> i</span>nclude</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> r</span>ead_from</span>:</span> e</span>nd</span> #</span> only new lines; use 'beginning' for backfill</span></span> </span> s</span>inks</span>:</span></span> s</span>3_logs</span>:</span></span> t</span>ype</span>:</span> a</span>ws_s3</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> b</span>ucket</span>:</span> m</span>y-logs</span></span> r</span>egion</span>:</span> e</span>u-central-1</span></span> #</span> Vector automatically appends a UUID to key_prefix — no explicit $UUID needed.</span></span> #</span> Multiple instances writing concurrently will never collide.</span></span> k</span>ey_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> c</span>ompression</span>:</span> g</span>zip</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span> #</span> raw lines — no JSON wrapping</span></span> b</span>atch</span>:</span></span> m</span>ax_bytes</span>:</span> 52428800</span> #</span> 50 MB</span></span> t</span>imeout_secs</span>:</span> 600</span> #</span> or 10 minutes</span></span></code></pre> For Azure Blob, replace the sink:</p> a</span>zure_blob_logs</span>:</span></span> t</span>ype</span>:</span> a</span>zure_blob</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> c</span>onnection_string</span>:</span> "</span>${AZURE_STORAGE_CONNECTION_STRING}</span>"</span></span> c</span>ontainer_name</span>:</span> n</span>ginx-logs</span></span> #</span> Same as S3: Vector appends a UUID automatically — uniqueness is built-in.</span></span> b</span>lob_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span></span> c</span>ompression</span>:</span> g</span>zip</span></span></code></pre> Filebeat → Logstash → S3🔗</a></h3> Filebeat</a> is the standard shipper in the Elastic Stack. It has an nginx module</a> that parses, enriches and structures the access log — but do not use the nginx module for AWFFull</strong>. The module converts log lines to JSON with split fields; AWFFull needs the raw Combined/CLF format.</p> Instead, use the generic filestream</code> input with no parsing:</p> #</span> /etc/filebeat/filebeat.yml</span></span> </span> f</span>ilebeat.inputs</span>:</span></span> -</span> t</span>ype</span>:</span> f</span>ilestream</span></span> i</span>d</span>:</span> n</span>ginx-access</span></span> p</span>aths</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> #</span> No parsers, no module — raw lines only</span></span> p</span>arsers</span>:</span> [</span>]</span></span> </span> o</span>utput.logstash</span>:</span></span> h</span>osts</span>:</span> [</span>"</span>logstash:5044</span>"</span>]</span></span></code></pre> Filebeat has no native S3 output. Use Logstash</strong> as the intermediary with the logstash-output-s3</code> plugin:</p> #</span> /etc/logstash/conf.d/nginx-s3.conf</span></span> </span> input </span>{</span></span> beats </span>{</span> port </span>=></span> 5044</span> }</span></span> }</span></span> </span> filter </span>{</span></span> #</span> No grok, no parsing — pass the raw message through unchanged</span></span> }</span></span> </span> output </span>{</span></span> s3 </span>{</span></span> bucket </span>=></span> "</span>my-logs</span>"</span></span> region </span>=></span> "</span>eu-central-1</span>"</span></span> prefix </span>=></span> "</span>nginx/%{+YYYY}/%{+MM}/%{+dd}/</span>"</span></span> codec </span>=></span> line </span>#</span> raw lines, one per file</span></span> encoding </span>=></span> "</span>gzip</span>"</span></span> size_file </span>=></span> 52428800</span> #</span> 50 MB per file</span></span> time_file </span>=></span> 10</span> #</span> or rotate after 10 minutes</span></span> #</span> Credentials from IAM role / environment</span></span> }</span></span> }</span></span></code></pre> The codec => line</code> setting is critical — it writes one log line per line to the file, which is exactly what AWFFull expects. The default json_lines</code> codec would break compatibility.</p> Note:</strong> Filebeat also supports shipping to Azure via Logstash with the logstash-output-azure_blob_storage</a> community plugin, using the same pattern.</p> Concurrent Write Safety🔗</a></h3> When multiple shipper instances write to the same bucket simultaneously (e.g. Kubernetes Pods with horizontal scaling, or multiple servers sending to one central bucket), the first question is: can this corrupt the data?</strong></p> Short answer: no — but data loss is possible if you don’t use unique keys.</strong></p> S3 and Azure Blob both guarantee atomic PUT semantics</strong> at the object level. A reader will always see either the complete previous version or the complete new version of an object — never a torn or half-written file. Byte-level corruption from concurrent writes is not possible.</p> The real risk is silent overwrite</strong>: two instances that generate the same object key will race, and the last writer wins. The first writer’s data is permanently lost.</p> Here is how the tools described above handle this:</p> Tool</th> Mechanism</th> Multiple instances</th></tr></thead> Fluent Bit → S3</td> $UUID</code> in s3_key_format</code></td> Safe — each instance generates a unique key</td></tr> Vector → S3</td> UUID appended automatically to key_prefix</code></td> Safe — built-in</td></tr> Vector → Azure Blob</td> UUID appended automatically to blob_prefix</code></td> Safe — built-in</td></tr> Logstash → S3</td> hostname + timestamp</code> in the generated filename</td> Mostly safe — risk only if two instances share an identical hostname and flush within the same second</td></tr> Fluent Bit → Azure Blob (append blob)</td> Append blob type with atomic server-side appends</td> Safe — Azure Blob Storage serialises concurrent appends to the same blob at the service level</td></tr> Fluent Bit → Azure Blob (block blob)</td> Path parameter sets the blob name</td> Unsafe if multiple instances use the same path</code> without a unique suffix</td></tr> </tbody></table> Practical rule:</strong> always include a UUID or a unique hostname token in your object key / blob path. All three tools support this out of the box; just avoid overriding their defaults with a static, shared key.</p> Log Loss at High-Traffic Sites🔗</a></h3> Concurrent write collisions are not the only way to lose log data. At high request rates — or under network pressure — the shipper itself can become a bottleneck. The common failure modes are:</p> 1. Memory buffer overflow</strong> Every shipper buffers incoming lines in memory while waiting to flush to S3/Azure. If the upload is slower than the ingestion rate (network hiccup, S3 throttle, slow endpoint), the buffer fills up. Once full, depending on configuration, the shipper either drops new lines silently</strong> or applies backpressure</strong> upstream — which can slow down the web server itself.</p> 2. Network outage or S3/Azure throttling</strong> Object storage APIs return 503 Slow Down</code> (S3) or 429 Too Many Requests</code> (Azure) under load. Without a persistent buffer and retry logic the shipper will drop records it cannot upload.</p> 3. Shipper crash without offset tracking</strong> If the shipper process dies with data still in its memory buffer, those records are gone. On restart it may resume from the wrong position: too early (duplicate records) or too late (gap in coverage).</p> 4. Log rotation race</strong> logrotate</code> (or nginx’s own rotation) can rename or truncate the access log before the shipper has finished reading it. The unread tail of the old file is silently lost.</p> Each tool has a different answer to these problems:</p> Tool</th> Persistence</th> At-least-once guarantee</th></tr></thead> Fluent Bit</td> Filesystem buffer (storage.type filesystem</code>)</td> Yes — survives restarts and network outages if filesystem buffering is enabled</td></tr> Vector</td> Disk buffer (type: disk</code> in buffer config)</td> Yes — survives restarts; default is in-memory only</td></tr> Logstash</td> Persistent queue (queue.type: persisted</code>)</td> Yes — survives restarts; default is in-memory only</td></tr> Filebeat</td> Registry file tracks read offset per file</td> Yes — resumes exactly where it left off after restart</td></tr> </tbody></table> Filebeat is the most robust for at-least-once delivery: it writes a registry file that records the exact byte offset and inode for every tailed file. After a crash or restart it picks up from that position. The one gap: if the log file is rotated and deleted</strong> before Filebeat reads to the end, the unread bytes are lost.</p> Recommendations for high-traffic sites:</strong></p> Enable filesystem / disk / persistent buffering in your shipper — never rely solely on in-memory buffers in production</li> Set a generous retry policy and back-off interval for S3/Azure upload errors</li> Configure log rotation with a delaycompress</code> / rotate_wait</code> window long enough for the shipper to finish reading the old file before it is compressed or deleted</li> Monitor the shipper’s own metrics (dropped records, buffer utilisation, upload errors) — these are separate from the web server metrics</li> </ul> Below are complete production-ready configurations with persistent buffering and retry logic enabled for each tool.</p> Fluent Bit — Filesystem Buffer + Retry🔗</a></h4> #</span> /etc/fluent-bit/fluent-bit.conf</span></span> </span> [</span>SERVICE</span>]</span></span> Flush 5</span></span> Daemon Off</span></span> Log_Level info</span></span> </span> #</span> Filesystem buffering — chunks are written to disk before upload.</span></span> #</span> Data survives Fluent Bit restarts and network outages up to the limit.</span></span> storage.path /var/lib/fluent-bit/buf/</span></span> storage.sync normal </span>#</span> fsync after each chunk write</span></span> storage.checksum off</span></span> storage.max_chunks_up 128 </span>#</span> max chunks in memory at once</span></span> </span> [</span>INPUT</span>]</span></span> Name tail</span></span> Path /var/log/nginx/access.log</span></span> Tag nginx.access</span></span> Read_from_Head false</span></span> Refresh_Interval 5</span></span> #</span> Offset database — Fluent Bit remembers the last read position.</span></span> #</span> Without this, every restart re-reads the file from the beginning.</span></span> DB /var/lib/fluent-bit/nginx-access.db</span></span> #</span> Use filesystem storage for this input's chunks (not memory)</span></span> storage.type filesystem</span></span> </span> [</span>OUTPUT</span>]</span></span> Name s3</span></span> Match nginx.access</span></span> bucket my-logs</span></span> region eu-central-1</span></span> s3_key_format /nginx/%Y/%m/%d/access_%H%M%S_$UUID.log.gz</span></span> compression gzip</span></span> total_file_size 50M</span></span> upload_timeout 10m</span></span> use_put_object On</span></span> </span> #</span> Retry: up to 5 attempts with exponential back-off (max ~160 s between tries).</span></span> #</span> Set False for unlimited retries (use with caution — can block the pipeline).</span></span> Retry_Limit 5</span></span> </span> #</span> Cap the on-disk buffer for this output at 1 GB.</span></span> #</span> When full, oldest chunks are dropped to make room.</span></span> storage.total_limit_size 1G</span></span></code></pre> The DB</code> file (SQLite) is the key to surviving log rotation: Fluent Bit stores the inode + offset, so even if logrotate</code> renames access.log</code> to access.log.1</code>, Fluent Bit will finish reading the old inode before switching to the new file.</p> Vector — Disk Buffer + Retry🔗</a></h4> #</span> /etc/vector/vector.yaml</span></span> </span> s</span>ources</span>:</span></span> n</span>ginx_access</span>:</span></span> t</span>ype</span>:</span> f</span>ile</span></span> i</span>nclude</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> r</span>ead_from</span>:</span> e</span>nd</span></span> #</span> Vector also writes a checkpoint file (similar to Filebeat's registry)</span></span> #</span> to /var/lib/vector/ by default — survives restarts automatically.</span></span> </span> s</span>inks</span>:</span></span> s</span>3_logs</span>:</span></span> t</span>ype</span>:</span> a</span>ws_s3</span></span> i</span>nputs</span>:</span> [</span>n</span>ginx_access</span>]</span></span> b</span>ucket</span>:</span> m</span>y-logs</span></span> r</span>egion</span>:</span> e</span>u-central-1</span></span> k</span>ey_prefix</span>:</span> "</span>nginx/{{ now() \| strftime('%Y/%m/%d') }}/</span>"</span></span> c</span>ompression</span>:</span> g</span>zip</span></span> e</span>ncoding</span>:</span></span> c</span>odec</span>:</span> t</span>ext</span></span> b</span>atch</span>:</span></span> m</span>ax_bytes</span>:</span> 52428800</span> #</span> 50 MB per object</span></span> t</span>imeout_secs</span>:</span> 600</span> #</span> or 10 minutes, whichever comes first</span></span> </span> #</span> Persistent disk buffer — events are written to disk before upload.</span></span> #</span> Survives Vector restarts; default is type: memory (lost on crash).</span></span> b</span>uffer</span>:</span></span> t</span>ype</span>:</span> d</span>isk</span></span> m</span>ax_size</span>:</span> 1073741824</span> #</span> 1 GB</span></span> w</span>hen_full</span>:</span> b</span>lock</span> #</span> apply backpressure instead of dropping events</span></span> </span> #</span> Retry policy for S3 upload failures (503, 429, network errors).</span></span> r</span>equest</span>:</span></span> r</span>etry_attempts</span>:</span> 10</span> #</span> max upload attempts per batch</span></span> r</span>etry_initial_backoff_secs</span>:</span> 1</span> #</span> first retry after 1 s</span></span> r</span>etry_max_duration_secs</span>:</span> 300</span> #</span> cap back-off at 5 minutes</span></span> t</span>imeout_secs</span>:</span> 60</span> #</span> per-request timeout</span></span> r</span>ate_limit_num</span>:</span> 100</span> #</span> max concurrent requests</span></span></code></pre> when_full: block</code> is the safe choice for log data — it slows the pipeline rather than silently dropping events. Use when_full: drop_newest</code> only if you explicitly accept loss under extreme backlog.</p> Filebeat + Logstash — Registry + Persistent Queue🔗</a></h4> Filebeat</strong> (/etc/filebeat/filebeat.yml</code>):</p> f</span>ilebeat.inputs</span>:</span></span> -</span> t</span>ype</span>:</span> f</span>ilestream</span></span> i</span>d</span>:</span> n</span>ginx-access</span></span> p</span>aths</span>:</span></span> -</span> /</span>var/log/nginx/access.log</span></span> p</span>arsers</span>:</span> [</span>]</span> #</span> no parsing — raw lines only</span></span> </span> #</span> Filebeat writes a registry to /var/lib/filebeat/registry/ that records</span></span> #</span> the inode, byte offset, and identifier for every tracked file.</span></span> #</span> On restart it resumes exactly where it stopped — no duplicates, no gaps.</span></span> f</span>ilebeat.registry.path</span>:</span> /</span>var/lib/filebeat/registry</span></span> </span> #</span> Internal memory queue — tune for throughput vs. latency.</span></span> #</span> These events are lost on crash; durability comes from the registry,</span></span> #</span> not from this queue.</span></span> q</span>ueue.mem</span>:</span></span> e</span>vents</span>:</span> 4096</span></span> f</span>lush.min_events</span>:</span> 512</span></span> f</span>lush.timeout</span>:</span> 5</span>s</span></span> </span> o</span>utput.logstash</span>:</span></span> h</span>osts</span>:</span> [</span>"</span>logstash:5044</span>"</span>]</span></span> #</span> Retry connecting to Logstash — Filebeat will keep trying indefinitely.</span></span> #</span> Events accumulate in the queue while Logstash is unreachable.</span></span> t</span>imeout</span>:</span> 30</span></span> m</span>ax_retries</span>:</span> 3</span> #</span> per batch; 0 = retry forever</span></span> b</span>ulk_max_size</span>:</span> 2048</span></span></code></pre> Logstash</strong> (/etc/logstash/logstash.yml</code>):</p> #</span> Persistent queue — events are written to disk between the input and filter/</span></span> #</span> output stages. Logstash survives restarts without losing in-flight events.</span></span> q</span>ueue.type</span>:</span> p</span>ersisted</span></span> q</span>ueue.max_bytes</span>:</span> 1</span>gb</span></span> q</span>ueue.path</span>:</span> /</span>var/lib/logstash/queue</span></span> </span> #</span> Checkpoint frequency — lower values = more durable, more I/O.</span></span> q</span>ueue.checkpoint.acks</span>:</span> 1024</span></span> q</span>ueue.checkpoint.writes</span>:</span> 1024</span></span> </span> #</span> Dead-letter queue — events that fail after all retries go here instead</span></span> #</span> of being silently dropped. Inspect with the dead_letter_queue input plugin.</span></span> d</span>ead_letter_queue.enable</span>:</span> true</span></span> d</span>ead_letter_queue.max_bytes</span>:</span> 1</span>gb</span></span> p</span>ath.dead_letter_queue</span>:</span> /</span>var/lib/logstash/dead_letter_queue</span></span></code></pre> Logstash pipeline</strong> (/etc/logstash/conf.d/nginx-s3.conf</code>):</p> input </span>{</span></span> beats </span>{</span> port </span>=></span> 5044</span> }</span></span> }</span></span> </span> filter </span>{</span></span> #</span> No grok, no parsing — pass the raw message through unchanged</span></span> }</span></span> </span> output </span>{</span></span> s3 </span>{</span></span> bucket </span>=></span> "</span>my-logs</span>"</span></span> region </span>=></span> "</span>eu-central-1</span>"</span></span> prefix </span>=></span> "</span>nginx/%{+YYYY}/%{+MM}/%{+dd}/</span>"</span></span> codec </span>=></span> line </span>#</span> one raw log line per line</span></span> encoding </span>=></span> "</span>gzip</span>"</span></span> size_file </span>=></span> 52428800</span> #</span> rotate object at 50 MB</span></span> time_file </span>=></span> 10</span> #</span> or after 10 minutes</span></span> #</span> Local staging directory — files are assembled here before upload.</span></span> #</span> Use a path on a real disk, not tmpfs, for crash safety.</span></span> temporary_directory </span>=></span> "</span>/var/lib/logstash/s3-tmp</span>"</span></span> #</span> Retry failed uploads before giving up and sending to the DLQ.</span></span> retry_count </span>=></span> 10</span></span> retry_delay </span>=></span> 1</span> #</span> seconds between retries</span></span> }</span></span> }</span></span></code></pre> With queue.type: persisted</code> and the dead-letter queue enabled, Logstash provides end-to-end at-least-once delivery: events survive a Logstash restart, and events that cannot be delivered after all retries land in the DLQ for manual inspection rather than being silently dropped.</p> Running AWFFull Against Object Storage Logs🔗</a></h2> </p> info </div> Info</strong></span> </div> The AWFFull container downloads the log files and the previous awffull.hist</code> state file from object storage, runs the analysis, then uploads the generated HTML + PNG reports and the updated history file back — no persistent volumes needed.</p> </div> </div> Once logs are stored with date-based prefixes, fetching a specific day is a single command:</p> From S3:</strong></p> #</span> Download all log files for 12 March 2026</span></span> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/12/</span> /tmp/logs/2026-03-12/</span></span> </span> #</span> Process with AWFFull (pass files in chronological order)</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>ls</span> /tmp/logs/2026-03-12/</span></span>.log.gz</span> \|</span> sort</span>)</span></span></code></pre> From Azure Blob:</strong></p> azcopy</span> copy</span> \</span></span> "</span>https://mystorageaccount.blob.core.windows.net/nginx-logs/nginx/2026/03/12/</span>"</span> \</span></span> /tmp/logs/2026-03-12/</span> \</span></span> -</span>-recursive</span></span> </span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>ls</span> /tmp/logs/2026-03-12/</span></span>.log.gz</span> \|</span> sort</span>)</span></span></code></pre> Downloading a full month</strong> (common for monthly reporting):</p> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/</span> /tmp/logs/2026-03/</span> -</span>-include</span> "</span>.gz</span>"</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> $(</span>find</span> /tmp/logs/2026-03/</span> -</span>name</span> "</span>.log.gz</span>"</span> \|</span> sort</span>)</span></span></code></pre> The sort</code> is important — AWFFull processes log records in the order it receives them. Files uploaded with timestamps in their names (e.g. access_0230_abc123.log.gz</code>) will sort correctly into chronological order automatically.</p> Running AWFFull in a Container World🔗</a></h2> AWFFull works perfectly well inside a container — the challenge is purely about where the log files come from and where the generated HTML goes. In a container-native environment there are several proven patterns:</p> Option 1 — Persistent Volume (classical approach)🔗</a></h3> The simplest lift-and-shift. Mount a persistent volume into the container for both the log input and the report output, exactly as you would with a local disk. Works identically to the bare-metal setup.</p> #</span> docker-compose example</span></span> s</span>ervices</span>:</span></span> a</span>wffull</span>:</span></span> i</span>mage</span>:</span> r</span>egistry.gitlab.com/aleks001/awfull:latest</span></span> v</span>olumes</span>:</span></span> -</span> /</span>var/log/nginx:/logs:ro</span> #</span> log files (read-only)</span></span> -</span> /</span>srv/awffull/reports:/reports</span> #</span> generated HTML + PNG output</span></span> -</span> /</span>etc/awffull.conf:/etc/awffull.conf:ro</span></span> c</span>ommand</span>:</span> [</span>"</span>-c</span>"</span>,</span> "</span>/etc/awffull.conf</span>"</span>,</span> "</span>/logs/access.log.gz</span>"</span>]</span></span></code></pre> In Kubernetes, replace the host-path mounts with a PersistentVolumeClaim</code>. A CronJob running once a night to process the previous day’s log rotation works very well here.</p> #</span> Kubernetes CronJob skeleton</span></span> a</span>piVersion</span>:</span> b</span>atch/v1</span></span> k</span>ind</span>:</span> C</span>ronJob</span></span> m</span>etadata</span>:</span></span> n</span>ame</span>:</span> a</span>wffull-monthly</span></span> s</span>pec</span>:</span></span> s</span>chedule</span>:</span> "</span>0 2 1 </span>"</span> #</span> 02:00 on the 1st of each month</span></span> j</span>obTemplate</span>:</span></span> s</span>pec</span>:</span></span> t</span>emplate</span>:</span></span> s</span>pec</span>:</span></span> c</span>ontainers</span>:</span></span> -</span> n</span>ame</span>:</span> a</span>wffull</span></span> i</span>mage</span>:</span> r</span>egistry.gitlab.com/aleks001/awfull:latest</span></span> a</span>rgs</span>:</span> [</span>"</span>-c</span>"</span>,</span> "</span>/etc/awffull.conf</span>"</span>,</span> "</span>/logs/access.log.gz</span>"</span>]</span></span> v</span>olumeMounts</span>:</span></span> -</span> n</span>ame</span>:</span> l</span>ogs mountPath</span>:</span> /</span>logs</span></span> -</span> n</span>ame</span>:</span> r</span>eports mountPath</span>:</span> /</span>reports</span></span> -</span> n</span>ame</span>:</span> c</span>onfig mountPath</span>:</span> /</span>etc/awffull.conf</span></span> s</span>ubPath</span>:</span> a</span>wffull.conf</span></span> v</span>olumes</span>:</span></span> -</span> n</span>ame</span>:</span> l</span>ogs persistentVolumeClaim</span>:</span> {</span> c</span>laimName</span>:</span> n</span>ginx-logs</span> }</span></span> -</span> n</span>ame</span>:</span> r</span>eports persistentVolumeClaim</span>:</span> {</span> c</span>laimName</span>:</span> a</span>wffull-reports</span> }</span></span> -</span> n</span>ame</span>:</span> c</span>onfig configMap</span>:</span> {</span> n</span>ame</span>:</span> a</span>wffull-config</span> }</span></span> r</span>estartPolicy</span>:</span> O</span>nFailure</span></span></code></pre> Trade-off:</strong> Simple and reliable, but you need to manage the PVC and expose the report directory via a web server separately (e.g. an nginx sidecar or a separate static-files pod).</p> Option 2 — S3 / S3-compatible Object Storage🔗</a></h3> This option covers two related questions:</p> Where do the generated reports go?</li> Can logs that were already shipped to S3 be fed back into AWFFull?</strong></li> </ol> The answer to both is yes — and combining them gives a fully cloud-native, serverless-friendly pipeline.</p> Shipping logs to S3</strong> is standard practice. Tools like Fluent Bit</a>, Vector</a>, or even a plain logrotate</code> + aws s3 cp</code> cron job send compressed log files to a bucket with a path like s3://my-logs/nginx/2026/03/01/access.log.gz</code>.</p> Reading them back</strong> for AWFFull is equally straightforward:</p> #!</span>/bin/sh</span></span> #</span> 1. Download history file (needed for incremental processing)</span></span> aws</span> s3</span> cp</span> s3://my-bucket/awffull/awffull.hist</span> /var/lib/awffull/</span> 2</span>></span>/dev/null</span> \|</span>\|</span> true</span></span> </span> #</span> 2. Download log files for the period to process</span></span> aws</span> s3</span> sync</span> s3://my-logs/nginx/2026/03/</span> /tmp/logs/</span> -</span>-include</span> "</span>.gz</span>"</span></span> </span> #</span> 3. Run AWFFull (reads logs, updates history, writes HTML)</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /tmp/logs/</span></span>.gz</span></span> </span> #</span> 4. Persist the updated history file back to S3</span></span> aws</span> s3</span> cp</span> /var/lib/awffull/awffull.hist</span> s3://my-bucket/awffull/</span></span> </span> #</span> 5. Sync reports to the reports bucket / static website</span></span> aws</span> s3</span> sync</span> /reports/</span> s3://my-reports-bucket/awffull/</span> -</span>-delete</span></span></code></pre> The awffull.hist</code> step is important: it is the only piece of persistent state AWFFull needs between runs. Without it, incremental processing (-p</code>) won’t know what was already counted. Storing it alongside the reports in the same bucket works well.</p> In Kubernetes, this entire sequence becomes a CronJob that downloads, processes, and re-uploads — no persistent volumes needed at all:</p> S3 (logs) ──► AWFFull container ──► S3 (reports + awffull.hist)</span></span> ▲</span></span> └── IAM role / workload identity</span></span></code></pre> Generate the report locally inside the container, then push the entire output directory to an S3 bucket. The bucket can be served directly as a static website (S3 static website hosting, CloudFront, MinIO, etc.).</p> #!</span>/bin/sh</span></span> #</span> entrypoint wrapper</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /logs/access.log.gz</span></span> </span> #</span> Sync output to S3</span></span> aws</span> s3</span> sync</span> /reports/</span> s3://my-bucket/awffull/</span> \</span></span> -</span>-delete</span> \</span></span> -</span>-content-type</span> "</span>text/html</span>"</span> \</span></span> -</span>-exclude</span> "</span>.png</span>"</span> \</span></span> -</span>-include</span> "</span>.html</span>"</span></span> </span> aws</span> s3</span> sync</span> /reports/</span> s3://my-bucket/awffull/</span> \</span></span> -</span>-exclude</span> "</span>.html</span>"</span> \</span></span> -</span>-include</span> "</span>.png</span>"</span></span></code></pre> In a container, add awscli</code> to the image or use a two-stage approach with an init container for awffull and a sidecar for the S3 sync. The AWS credentials come from an IAM role (EKS), a Kubernetes secret, or environment variables.</p> Trade-off:</strong> Zero storage management, instant global CDN distribution, built-in versioning. Requires network access from the runner and appropriate IAM/bucket permissions.</p> Option 3 — Azure Blob Storage🔗</a></h3> Same pattern as S3, using the Azure CLI (az storage blob</code>) or azcopy</code>. The output directory is synced to a container within a Storage Account, which can be configured as a static website endpoint.</p> Just like S3, logs that were already shipped to Azure Blob Storage (e.g. via Fluent Bit’s Azure Blob output, Diagnostic Settings, or a custom pipeline) can be downloaded and fed directly to AWFFull:</p> #!</span>/bin/sh</span></span> STORAGE</span>=</span>"</span>https://<storage-account>.blob.core.windows.net</span>"</span></span> </span> #</span> 1. Download history file</span></span> azcopy</span> copy</span> "</span>$</span>{</span>STORAGE</span>}</span>/awffull/awffull.hist</span>"</span> /var/lib/awffull/</span> 2</span>></span>/dev/null</span> \|</span>\|</span> true</span></span> </span> #</span> 2. Download log files for the period</span></span> azcopy</span> copy</span> "</span>$</span>{</span>STORAGE</span>}</span>/nginx-logs/2026/03/</span>"</span> /tmp/logs/</span> -</span>-recursive</span></span> </span> #</span> 3. Run AWFFull</span></span> awffull</span> -</span>c</span> /etc/awffull.conf</span> /tmp/logs/</span></span>.gz</span></span> </span> #</span> 4. Persist history back to Blob</span></span> azcopy</span> copy</span> /var/lib/awffull/awffull.hist</span> "</span>$</span>{</span>STORAGE</span>}</span>/awffull/awffull.hist</span>"</span></span> </span> #</span> 5. Sync reports to static website container ($web)</span></span> azcopy</span> sync</span> /reports/</span> "</span>$</span>{</span>STORAGE</span>}</span>/</span>\$</span>web/awffull/</span>"</span> \</span></span> -</span>-recursive</span> \</span></span> -</span>-delete-destination=true</span></span></code></pre> Authentication uses a managed identity (AKS workload identity), a SAS token, or a service principal — no credentials baked into the image:</p> #</span> With workload identity (AKS)</span></span> az</span> login</span> -</span>-identity</span></span> az</span> storage</span> blob</span> sync</span> \</span></span> -</span>-source</span> /reports/</span> \</span></span> -</span>-container</span> '</span>$web</span>'</span> \</span></span> -</span>-account-name</span> <</span>storage-accoun</span>t</span>></span> \</span></span> -</span>-destination</span> awffull/</span></span></code></pre> Enable the Static website</strong> feature on the Storage Account and the reports are immediately reachable at https://<storage-account>.z6.web.core.windows.net/awffull/index.html</code>.</p> Trade-off:</strong> Native Azure integration, no separate web server needed, HTTPS included. Requires AKS workload identity or explicit credentials.</p> Choosing a Pattern🔗</a></h3> Scenario</th> Recommended approach</th></tr></thead> Simple on-prem Kubernetes</td> Persistent Volume + nginx sidecar</td></tr> AWS / EKS</td> S3 + CloudFront static website</td></tr> Azure / AKS</td> Azure Blob static website + workload identity</td></tr> Self-hosted, no cloud</td> S3 sync</td></tr> CI/CD pipeline artifact</td> Upload as GitLab/GitHub artifact, expire in 90 days</td></tr> </tbody></table> In all cases the AWFFull container itself remains stateless and ephemeral — it runs, generates output, uploads it, and exits. The history file (awffull.hist</code>) is the only piece of persistent state needed for incremental processing; store it on the same volume or object storage path.</p> AWFFull is free software licensed under GPL v3 or later.</em> The project is at https://gitlab.com/aleks001/awfful</a>.</em></p> loadgen-rs (Short): Results, Quick Start, and Tool Choice 2026-03-03T00:00:00+00:00 This is the short version. If you want profiling details, FFI internals, and full cloud deployment walkthroughs, read the full article: loadgen-rs: An HTTP Benchmark Client in Rust</a>.</p> TL;DR🔗</a></h2> loadgen-rs</code> is a Rust benchmark client for HTTP/1.1, HTTP/2, and HTTP/3 with:</p> one CLI for all protocols</li> JSONL/CSV output for automation</li> distributed controller/worker mode</li> scripted Deno scenarios via FFI (k6-style checks/extractors)</li> </ul> In short: very strong for reproducible automation and multi-node testing. If your only goal is absolute H3 peak throughput, h2load</code> still has an edge.</p> Quick Start🔗</a></h2> Single-node cli🔗</a></h3> #</span> H2 example, 10s run</span></span> loadgen-rs</span> -</span>n</span> 0</span> -</span>D</span> 10s</span> -</span>c</span> 8</span> -</span>t</span> 4</span> -</span>m</span> 10</span> \</span></span> -</span>-alpn-list=h2</span> -</span>-insecure</span> '</span>https://bench.local:8082/?s=256k</span>'</span></span></code></pre>Single-node container🔗</a></h3> #</span> HTTP/3 (QUIC) 10 seconds</span></span> podman</span> run</span> -</span>it</span> -</span>-rm</span> \</span></span> -</span>-network</span> host</span> \</span></span> -</span>-name</span> gh-loadgen</span> ghcr.io/git001/loadgen-rs:v0.3.0</span> \</span></span> -</span>D</span> 10s</span> -</span>c</span> 8</span> -</span>t</span> 4</span> -</span>m</span> 10</span> \</span></span> -</span>-alpn-list=h3</span> \</span></span> -</span>-insecure</span> \</span></span> '</span>https://bench.local:8082/?s=256k</span>'</span></span></code></pre>Distributed🔗</a></h3> #</span> 1) start workers</span></span> bash</span> examples/worker-start.sh</span></span> </span> #</span> 2) run controller</span></span> deno</span> run</span> -</span>-allow-ffi</span> -</span>-allow-net</span> -</span>-allow-read</span> -</span>-allow-env</span> \</span></span> examples/distributed.ts</span> http://worker1:9091</span> http://worker2:9091</span></span></code></pre> Why distributed matters: you can scale beyond one machine and still get statistically correct merged percentiles (histogram merge, not naive p99 averaging).</p> Benchmark Snapshot vs h2load🔗</a></h2> High-concurrency test (-c 512 -t 8 -m 8</code>, 30s):</p> Protocol</th> h2load RPS</th> loadgen-rs RPS</th> Ratio</th></tr></thead> H1</td> 18,851</td> 19,148</td> 101.6%</td></tr> H2</td> 17,837</td> 18,025</td> 101.1%</td></tr> H3</td> 9,006</td> 6,702</td> 74.4%</td></tr> </tbody></table> Takeaway:</p> H1/H2 throughput: effectively on par (or slightly better for loadgen-rs in this run).</li> H3 throughput: h2load remains faster.</li> Memory footprint: h2load is smaller; loadgen-rs typically used more memory in tests.</li> </ul> Which Tool Should You Use?🔗</a></h2> Need</th> Better fit</th></tr></thead> Highest H3 throughput only</td> h2load</td></tr> One tool for H1/H2/H3 + machine-readable reports</td> loadgen-rs</td></tr> Distributed multi-machine load generation</td> loadgen-rs</td></tr> Scripted scenarios with checks/extractors and correlation</td> loadgen-rs</td></tr> Lowest memory footprint</td> h2load</td></tr> </tbody></table> Next Step🔗</a></h2> For the complete deep dive (profiling, architecture, FFI API design, Terraform/Ansible workflow), continue with the full article: loadgen-rs: An HTTP Benchmark Client in Rust</a>.</p> loadgen-rs: An HTTP Benchmark Client in Rust 2026-03-03T00:00:00+00:00 TL;DR / Executive Summary🔗</a></h2> Prefer a shorter read first? loadgen-rs (Short): Results, Quick Start, and Tool Choice</a></p> loadgen-rs is a Rust HTTP benchmark client that covers HTTP/1.1, HTTP/2, and HTTP/3 in one tool, outputs machine-readable JSONL/CSV, and scales from single-node runs to distributed multi-worker tests.</p> H1/H2 throughput is competitive with h2load, while H3 still favors h2load in peak throughput.</li> loadgen-rs is strong when you want automation-friendly output and one consistent workflow across protocols.</li> Distributed mode is a first-class feature, not an afterthought: controller + workers + correct histogram merge.</li> A Deno/FFI layer enables scripted, k6-style scenarios with checks, extractors, and correlation.</li> </ul> What Is loadgen-rs?🔗</a></h2> Why I Built It🔗</a></h3> If you benchmark HTTP servers, you probably know h2load</a> from the nghttp2 project. It’s a very solid and mature tool, but there are a few pain points:</p> HTTP/3 support is not enabled by default</strong> — h2load can do H3/QUIC, but only via a separate build variant with ngtcp2.</li> Requires a C compiler and many native libraries</strong> — this makes source builds and CI setup more cumbersome, especially across different environments.</li> </ul> I wanted a tool that tests all three HTTP protocols and outputs the results directly as JSON and easy</em> to build. So I built loadgen-rs</a></strong>.</p> Core Capabilities (H1/H2/H3, JSONL, Distributed, Scripting)🔗</a></h3> Component</th> Crate</th> Why</th></tr></thead> Runtime</td> tokio (multi-thread)</td> De-facto standard for async Rust</td></tr> HTTP/1.1</td> Raw TCP + httparse</td> Maximum control, zero framework overhead</td></tr> HTTP/2</td> h2 (direct, no hyper)</td> Avoids hyper’s abstraction layer</td></tr> TLS</td> rustls (ring backend)</td> Pure Rust, no OpenSSL dependency hell</td></tr> HTTP/3</td> quinn + h3 + h3-quinn</td> QUIC stack in Rust</td></tr> CLI</td> clap (derive)</td> Ergonomic, type-safe</td></tr> Metrics</td> hdrhistogram</td> Industry standard for latency measurements</td></tr> Allocator</td> mimalloc</td> Better multi-threaded allocation performance</td></tr> </tbody></table> An early design choice was to bypass hyper</a> entirely. For H1, we use raw TCP sockets with httparse</code> for response parsing. For H2, we use the h2</code> crate directly. This eliminates one abstraction layer and gives full control over connection management, flow control tuning, and request templating. Hyper</a> is a great library, but its abstraction model didn’t fit the needs of this benchmarking tool.</p> In addition to single-node CLI benchmarking, loadgen-rs includes distributed controller/worker execution and scripted scenario support through a native FFI layer for Deno.</p> Runtime and Execution Model🔗</a></h3> </figure> Connection Strategy🔗</a></h4> The -c</code> and -m</code> flags follow h2load semantics:</p> -c N</code></strong> spawns N parallel connections distributed across worker threads</li> -m N</code></strong> limits concurrent in-flight requests per connection (H2/H3: concurrent streams; H1: in-flight cap)</li> Total parallelism: c m</code></li> </ul> Each worker thread runs its own current_thread</code> Tokio runtime — no cross-thread synchronization in the hot path</strong>. Workers own local HdrHistograms that are merged only at the end.</p> Count vs. Duration Mode🔗</a></h4> Count mode</strong> (-n 1000</code>): A shared AtomicU64</code> budget that workers decrement via CAS</li> Duration mode</strong> (-D 10s</code>): A CancellationToken</code> that fires after the specified duration</li> -D</code> takes priority over -n</code> (matching h2load behavior)</li> </ul> RPS Semantics (h2load-compatible)🔗</a></h4> --rps</code> is implemented as a per-client</strong> request start rate, matching h2load</code>.</p> Per-client target: --rps</code></li> Aggregate target: --rps * -c</code></li> </ul> This matters for comparisons:</p> -c 10 --rps 20</code> targets roughly 200</code> started req/s overall</li> -c 10 --rps 200</code> targets roughly 2000</code> started req/s overall</li> </ul> Also note: in short count mode</strong> runs (-n</code>), achieved %</code> can appear above or below 100% because startup and shutdown overhead dominate a short elapsed window. For stable pacing comparisons, use duration mode</strong>.</p> Output and TLS Notes🔗</a></h3> The JSONL output contains everything a Python script needs:</p> {</span></span> "</span>proto</span>"</span>:</span> "</span>h2</span>"</span>,</span></span> "</span>url</span>"</span>:</span> "</span>https://target:8443/</span>"</span>,</span></span> "</span>clients</span>"</span>:</span> 8</span>,</span></span> "</span>threads</span>"</span>:</span> 4</span>,</span></span> "</span>rps</span>"</span>:</span> 12543.7</span>,</span></span> "</span>latency_p50_us</span>"</span>:</span> 320</span>,</span></span> "</span>latency_p90_us</span>"</span>:</span> 890</span>,</span></span> "</span>latency_p99_us</span>"</span>:</span> 2100</span>,</span></span> "</span>status_counts</span>"</span>:</span> {</span>"</span>200</span>"</span>:</span> 125437</span>}</span>,</span></span> "</span>status_2xx</span>"</span>:</span> 125437</span>,</span></span> "</span>err_total</span>"</span>:</span> 0</span>,</span></span> "</span>elapsed_s</span>"</span>:</span> 10.001</span></span> }</span></span></code></pre> In Python:</p> import</span> json</span></span> results</span> =</span> [</span>json</span>.</span>loads</span>(</span>line</span>)</span> for</span> line</span> in</span> open</span>(</span>"</span>results.jsonl</span>"</span>)</span>]</span></span> df</span> =</span> pd</span>.</span>DataFrame</span>(</span>results</span>)</span></span> df</span>.</span>plot</span>(</span>x</span>=</span>"</span>clients</span>"</span>,</span> y</span>=</span>"</span>rps</span>"</span>,</span> kind</span>=</span>"</span>bar</span>"</span>)</span></span></code></pre>TLS Configuration🔗</a></h4> --insecure</code> implements a custom ServerCertVerifier</code> that accepts everything. --tls-ciphers</code> allows selecting specific cipher suites — with validation that H3 only permits TLS 1.3 ciphers. Custom CA certificates can be loaded via --tls-ca</code>.</p> Quick Start🔗</a></h2> Single-Node Benchmark in 30 Seconds (cli)🔗</a></h3> #</span> HTTP/1.1, 1000 requests</span></span> loadgen-rs</span> -</span>n</span> 1000</span> -</span>c</span> 4</span> -</span>t</span> 2</span> -</span>-h1</span> '</span>http://bench.local:8081/?s=256k</span>'</span></span> </span> #</span> HTTPS</span></span> loadgen-rs</span> -</span>n</span> 1000</span> -</span>c</span> 4</span> -</span>t</span> 2</span> -</span>-h1</span> -</span>-insecure</span> '</span>https://bench.local:8082/?s=256k</span>'</span></span> </span> #</span> HTTP/2, 10 seconds</span></span> loadgen-rs</span> -</span>n</span> 0</span> -</span>D</span> 10s</span> -</span>c</span> 8</span> -</span>t</span> 4</span> -</span>m</span> 10</span> \</span></span> -</span>-alpn-list=h2</span> -</span>-insecure</span> '</span>https://bench.local:8082/?s=256k</span>'</span></span> </span> #</span> HTTP/3 (QUIC)</span></span> loadgen-rs</span> -</span>n</span> 0</span> -</span>D</span> 10s</span> -</span>c</span> 4</span> -</span>t</span> 2</span> \</span></span> -</span>-alpn-list=h3</span> -</span>-insecure</span> '</span>https://bench.local:8082/?s=256k</span>'</span></span></code></pre>Single-Node Benchmark in 30 Seconds (container)🔗</a></h3> If you don’t have podman</code>, you can use docker</code> instead.</p> #</span> HTTP/3 (QUIC) 10 seconds</span></span> podman</span> run</span> -</span>it</span> -</span>-rm</span> \</span></span> -</span>-network</span> host</span> \</span></span> -</span>-name</span> gh-loadgen</span> ghcr.io/git001/loadgen-rs:v0.3.0</span> \</span></span> -</span>D</span> 10s</span> -</span>c</span> 8</span> -</span>t</span> 4</span> -</span>m</span> 10</span> \</span></span> -</span>-alpn-list=h3</span> \</span></span> -</span>-insecure</span> \</span></span> '</span>https://bench.local:8082/?s=256k</span>'</span></span></code></pre>Distributed Benchmark in 2 Minutes🔗</a></h3> #</span> Start workers (one per machine, or locally for testing)</span></span> bash</span> examples/worker-start.sh</span></span> </span> #</span> Run controller</span></span> deno</span> run</span> -</span>-allow-ffi</span> -</span>-allow-net</span> -</span>-allow-read</span> -</span>-allow-env</span> \</span></span> examples/distributed.ts</span> http://worker1:9091</span> http://worker2:9091</span></span></code></pre> Or programmatically:</p> import</span> {</span> Controller</span> }</span> from</span> "</span>./ts/mod.ts</span>"</span>;</span></span> </span> const</span> controller</span> =</span> new</span> Controller</span>(</span>{</span></span> workers</span>:</span> [</span>"</span>http://worker1:9091</span>"</span>,</span> "</span>http://worker2:9091</span>"</span>]</span>,</span></span> config</span>:</span> {</span></span> url</span>:</span> "</span>https://target:8443/</span>"</span>,</span></span> protocol</span>:</span> "</span>h2</span>"</span>,</span></span> clients</span>:</span> 200</span>,</span> //</span> split: 100 per worker</span></span> threads</span>:</span> 4</span>,</span></span> duration_s</span>:</span> 30</span>,</span></span> insecure</span>:</span> true</span>,</span></span> }</span>,</span></span> }</span>)</span>;</span></span> </span> const</span> result</span> =</span> await</span> controller</span>.</span>run</span>(</span>)</span>;</span></span> console</span>.</span>log</span>(</span>result</span>.</span>merged_report</span>.</span>rps</span>)</span>;</span></span></code></pre>Minimal Comparison Against h2load🔗</a></h3> #</span> 1) H1 count mode (same request budget)</span></span> h2load</span> -</span>-h1</span> -</span>n</span> 200</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-h1</span> -</span>n</span> 200</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients * 20 rps/client)</span></span> h2load</span> -</span>-h1</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-h1</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 3) H2 duration mode with pacing (same effective target)</span></span> h2load</span> -</span>-alpn-list=h2</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-alpn-list</span> h2</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 4) H3 duration mode with pacing (same effective target)</span></span> h2load</span> -</span>-alpn-list=h3</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-alpn-list</span> h3</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span></code></pre>Key Results vs h2load🔗</a></h2> After implementing and tuning the tool, I ran systematic A/B comparisons against h2load</code> on the same hardware and target server.</p> High-Concurrency Benchmark🔗</a></h3> Setup:</p> Duration: 30 seconds per run, 30-second cooldown between runs</li> Configuration: -c 512 -t 8 -m 8</code></li> Target: https://127.0.0.1:8082/?s=256k</code> (256 KB response body)</li> h2load runs via podman run --rm --network host localhost/h2load</code></li> </ul> Protocol</th> h2load RPS</th> loadgen-rs RPS</th> Ratio</th> h2load Latency (mean)</th> loadgen-rs Latency (mean)</th></tr></thead> H1</strong></td> 18,851</td> 19,148</strong></td> 101.6%</strong></td> 27.07 ms</td> 25.69 ms</td></tr> H2</strong></td> 17,837</td> 18,025</strong></td> 101.1%</strong></td> 125.59 ms</td> 223.91 ms</td></tr> H3</strong></td> 9,006</strong></td> 6,702</td> 74.4%</td> 450.15 ms</td> 604.17 ms</td></tr> </tbody></table> Key observations:</p> H1</strong>: loadgen-rs is faster</strong> than h2load on both throughput and latency. The raw TCP + httparse approach with pre-built request templates pays off.</li> H2</strong>: Throughput is essentially identical. However, mean latency is ~1.8x higher in loadgen-rs due to an architectural difference (see profiling section below).</li> H3</strong>: h2load (using ngtcp2/nghttp3 in C) retains a ~25% throughput advantage over quinn/h3 in Rust.</li> </ul> To be fair, h2load was sometimes better than loadgen-rs, and vice versa.</p> Resource Usage (CPU and Memory)🔗</a></h3> For resource fairness, both tools were run as containers and sampled with podman stats</code>:</p> Setup:</p> 3 repeats per point, duration=5s</code>, c=4</code>, t=2</code>, m in {1,8}</code></li> </ul> Metric</th> H1/H2</th> H3</th></tr></thead> CPU time</td> mostly +2%</code> to +8%</code></td> -1%</code> to +10%</code></td></tr> Peak memory</td> about +410%</code> to +914%</code></td> about +1393%</code></td></tr> </tbody></table> In absolute terms, peak memory was typically:</p> h2load</code>: around 2.3 MB</code> to 3.0 MB</code></li> loadgen-rs</code>: around 15 MB</code> to 39 MB</code></li> </ul> The higher memory usage comes from Tokio’s runtime overhead, per-worker HdrHistograms, and Rust’s buffered I/O layers. In practice, even 39 MB is negligible for a benchmark client.</p> Which Tool Should You Choose?🔗</a></h2> Decision Matrix by Use Case🔗</a></h3> Priority</th> Recommendation</th></tr></thead> H1/H2 throughput + automation</td> loadgen-rs</strong> — matches or beats h2load, with native JSONL output</td></tr> H3 peak throughput</td> h2load</strong> — ngtcp2 is ~25% faster than quinn</td></tr> Lowest memory footprint</td> h2load</strong> — 3 MB vs 15-39 MB</td></tr> Cross-protocol consistency</td> loadgen-rs</strong> — same tool, same output format for H1/H2/H3</td></tr> Multi-machine load generation</td> loadgen-rs</strong> — built-in distributed mode with correct histogram merge</td></tr> Cloud-provisioned benchmarks</td> loadgen-rs</strong> — Terraform + Ansible pipeline for Hetzner Cloud</td></tr> Distributed scripted scenarios</td> loadgen-rs</strong> — workers run multi-step checks/extractors via POST /run-script</code></td></tr> </tbody></table> Practical Recommendation🔗</a></h3> For most benchmarking workflows, the choice comes down to this: do you need the absolute highest H3 throughput, or do you want a single binary with all protocols available out of the box?</p> Distributed Mode Overview🔗</a></h2> A single loadgen-rs process can saturate most test targets, but sometimes you need more — testing a load balancer with geographically distributed traffic, generating more requests than one machine’s NIC can handle, or simply simulating a more realistic multi-source access pattern.</p> Architecture🔗</a></h3> The distributed mode uses a TypeScript controller (running in Deno) that orchestrates multiple worker-agents over plain HTTP/JSON — no gRPC or custom protocol needed:</p> </figure> Each worker is a lightweight Deno HTTP server (~120 LOC) that wraps the same LoadgenFFI</code> batch API used for single-node benchmarks. The controller splits the total client count (-c</code>) evenly across workers, sets a coordinated start timestamp, and fires all jobs concurrently.</p> The histogram merge problem🔗</a></h3> Merging latency results from distributed workers is deceptively tricky. The naive approach — averaging p99 values from each worker — is statistically wrong</strong>. If worker A has p99 = 5ms (from 50K requests) and worker B has p99 = 15ms (from 50K requests), the combined p99 is not</em> 10ms. It depends on the full distribution shape.</p> The correct approach is to merge the underlying histograms. Each worker serializes its HdrHistogram using V2-Deflate format (a compact binary encoding, typically 1–5 KB per histogram), base64-encodes it, and includes it in the RunReport</code> JSON:</p> {</span></span> "</span>rps</span>"</span>:</span> 136949.0</span>,</span></span> "</span>latency_p99_us</span>"</span>:</span> 210</span>,</span></span> "</span>latency_hist_b64</span>"</span>:</span> "</span>HISTFAAAB...</span>"</span>,</span></span> "</span>ttfb_hist_b64</span>"</span>:</span> "</span>HISTFAAAB...</span>"</span>,</span></span> "</span>connect_hist_b64</span>"</span>:</span> "</span>HISTFAAAB...</span>"</span></span> }</span></span></code></pre> The controller passes all worker reports to loadgen_merge_reports()</code> — a new FFI export that deserializes each histogram, merges them with Histogram::add()</code>, and recomputes all percentile fields from the combined distribution. This is the same merge algorithm used internally for per-thread metrics within a single process.</p> Coordinated start🔗</a></h3> For benchmark results to be meaningful, all workers should start at the same time. The controller calculates start_at = now + 500ms</code>, includes it in each worker’s /run</code> request, and each worker sleeps until that timestamp before starting the benchmark. Clock skew under 10ms is acceptable for benchmark purposes — and on a local network or the same machine, skew is typically sub-millisecond.</p> Minimal new code, maximum reuse🔗</a></h3> The implementation reuses the existing FFI infrastructure almost entirely:</p> Component</th> New code</th> Reused</th></tr></thead> Rust: merge_distributed_reports()</code></td> ~120 LOC</td> Existing merge_metrics()</code> pattern</td></tr> Rust: FFI export</td> ~25 LOC</td> Existing catch_unwind</code> + error slot pattern</td></tr> TypeScript: worker-agent</td> ~120 LOC</td> LoadgenFFI</code> class</td></tr> TypeScript: Controller</td> ~150 LOC</td> mergeReports()</code> FFI wrapper</td></tr> </tbody></table> Only two new Rust dependencies were needed: base64</code> (for histogram serialization) was already an indirect dependency via reqwest. The export_histograms</code> field in BenchConfig</code> defaults to false</code>, so the existing CLI and single-node FFI path are completely unaffected.</p> Real-world validation🔗</a></h3> Testing 2 local workers against the same target (H2, 8 total clients, 5s):</p> Setup</th> RPS</th> p99 Latency</th></tr></thead> Single-node (-c 8 -m 4</code>)</td> 284,226</td> 185 us</td></tr> Distributed (2 × 4 clients)</td> 269,351</td> 207 us</td></tr> Ratio</td> 94.8%</strong></td> </td></tr> </tbody></table> The ~5% overhead comes from the HTTP/JSON coordination round-trip and the coordinated start delay. In a real multi-machine deployment (where each worker has its own NIC and CPU), total throughput scales linearly — you’re no longer bound by a single machine’s resources.</p> Scripted Scenarios Overview (Deno / k6-style)🔗</a></h2> After the initial release, I added a TypeScript/Deno layer on top of the native FFI to make scripted testing and k6-style output easier to compare.</p> What was added🔗</a></h3> A typed Deno SDK in ts/</code>: LoadgenFFI</code> for run-level benchmarking</li> LoadgenStepFFI</code> for step/session execution</li> </ul> </li> A reusable printK6LikeSummary()</code> formatter used by the Deno examples</li> New comparison examples: examples/ab-compare.ts</code> and examples/k6/ab-compare.ts</code> (closed model)</li> examples/ab-compare-rps.ts</code> and examples/k6/ab-compare-rps.ts</code> (fixed RPS)</li> </ul> </li> A one-command comparison helper: scripts/ab_compare_report.sh</code> (prints req/s, avg, p99, fail rate, H2 check)</li> </ul> </li> </ul> Important fairness fix🔗</a></h3> One easy mistake is RPS interpretation:</p> In loadgen-rs, --rps</code> is per client</strong></li> Aggregate target is rps_per_client * clients</code></li> </ul> The Deno comparison script now maps a total target RPS to per-client RPS automatically (RPS / CLIENTS</code>) to match k6 constant-arrival tests.</p> Why latency can differ even at similar req/s🔗</a></h3> Even when request rate is aligned, latency can diverge if concurrency shape is not aligned:</p> number of active clients/VUs</li> streams per connection (-m</code> / max_streams</code>)</li> thread count and connection reuse behavior</li> </ul> In short: equal target RPS does not automatically imply equal queueing behavior.</p> Stage profile support (k6-like)🔗</a></h3> I also added examples/ab-stages.ts</code>, which emulates stage ramps by splitting stages into short segments and running sequential FFI runs. This gives a practical equivalent for basic k6 stage workflows.</p> Correlation path🔗</a></h3> For dynamic data and sequence scenarios, runScriptScenario()</code> supports:</p> execution_mode: "fetch"</code> (pure Deno)</li> execution_mode: "ffi-step"</code> (native step transport)</li> </ul> The step API design and contract are documented in:</p> docs/script-mode-ffi.md</code></li> docs/ffi-step-api.md</code></li> docs/ffi-contract.md</code></li> </ul> k6 Parity Snapshot (as of March 1, 2026)🔗</a></h3> Short answer: Deno + loadgen-rs is now close to k6 for HTTP-focused scripted workflows, but it is not yet a full 1:1 replacement.</p> What is now very similar:</p> Scripted scenarios with step chains, checks, and extractors</li> Correlation and dynamic data ({{var}}</code> templates, JSON/regex/header/DOM extract)</li> Cookie handling and redirect policies</li> Two execution paths: pure fetch</code> and native ffi-step</code></li> Multi-protocol support in native step mode (h1</code>, h2</code>, h3</code>)</li> k6-like terminal summary output and side-by-side comparison scripts</li> </ul> Where k6 is still stronger:</p> More mature executor/scenario system overall (arrival/vu/stage/ramping breadth)</li> Larger built-in ecosystem (k6/http</code>, ws/browser modules, extension landscape)</li> Richer output/integration stack (cloud and common observability backends)</li> Longer production track record for large shared test suites and CI pipelines</li> </ul> Practical takeaway:</p> If your priority is HTTP load + correlation + native H1/H2/H3 coverage with comparable CLI summaries, Deno + loadgen-rs is already highly usable. If you need the full k6 orchestration and ecosystem depth, k6 still has an edge.</p> The distributed mode isn’t limited to raw throughput benchmarks. Workers also support POST /run-script</code> — the same scripted scenario engine used locally, but executed remotely on each worker. This means you can run multi-step request sequences with checks, extractors, cookies, and response validation across a fleet of machines.</p> The implementation required minimal changes: the worker-agent imports runScriptScenario()</code> from the existing script_mode.ts</code> and exposes it as a new HTTP endpoint. The controller sends a ScriptScenarioConfig</code> (same type used for local script mode) to each worker and collects ScriptRunResult</code> objects with per-check pass/fail counters.</p> A practical example ships as examples/html-check.ts</code> — a verification script that tests whether each worker can reach a target URL and parse the response correctly:</p> const</span> config</span>:</span> ScriptScenarioConfig</span> =</span> {</span></span> vus</span>:</span> 3</span>,</span></span> duration_s</span>:</span> 5</span>,</span></span> execution_mode</span>:</span> "</span>ffi-step</span>"</span>,</span></span> step_session_config</span>:</span> {</span></span> protocol</span>:</span> "</span>h2</span>"</span>,</span></span> insecure</span>:</span> true</span>,</span></span> response_headers</span>:</span> true</span>,</span></span> response_body_limit</span>:</span> 2_000_000</span>,</span></span> }</span>,</span></span> steps</span>:</span> [</span>{</span></span> name</span>:</span> "</span>html-get</span>"</span>,</span></span> method</span>:</span> "</span>GET</span>"</span>,</span></span> url</span>:</span> targetUrl</span>,</span></span> capture_body</span>:</span> true</span>,</span></span> checks</span>:</span> {</span></span> status_in</span>:</span> [</span>200</span>,</span> 403</span>]</span>,</span></span> body_includes</span>:</span> [</span>"</span><title>Example</title></span>"</span>]</span>,</span></span> header_includes</span>:</span> {</span> "</span>set-cookie</span>"</span>:</span> "</span>session=</span>"</span> }</span>,</span></span> }</span>,</span></span> }</span>]</span>,</span></span> }</span>;</span></span></code></pre> The controller sends this config to each worker via POST /run-script</code>, then aggregates the results into a per-worker table. If the target returns both the HTML element and the cookie on every single response, the iteration count, HTML match count, and header match count should be identical — any mismatch indicates a parsing or check evaluation bug:</p> === per-worker results ===</span></span> </span> worker iterations html_match header_match ok?</span></span> -------------------------------------------------------------------------------------</span></span> http://worker1:9091 62 62 62 PASS</span></span> http://worker2:9091 47 47 47 PASS</span></span> -------------------------------------------------------------------------------------</span></span> TOTAL 109 109 109 PASS</span></span></code></pre> All counts match perfectly across both workers. This validates the full chain: Terraform-provisioned infrastructure, Ansible-deployed workers, ffi-step</code> execution mode with reqwest on the Rust side, response header capture, and the check evaluation engine — all working end-to-end on remote machines.</p> The header_includes</code> check type was added specifically for this use case. Unlike header_exists</code> (which only checks presence) or header_equals</code> (which requires an exact match), header_includes</code> tests whether a header value contains</em> a substring — essential for Set-Cookie</code> headers where the full value includes expiry, path, and other attributes beyond the cookie name.</p> Multi-Step Login/Logout Workflow🔗</a></h4> The scripted scenario engine really shines when steps depend on each other. A second distributed example — examples/roundcube-login.ts</code> — verifies a full Roundcube Webmail login/logout cycle across workers. Unlike the single-step HTML check, this scenario chains three HTTP requests with state flowing between them:</p> steps</span>:</span> [</span></span> {</span></span> name</span>:</span> "</span>login-page</span>"</span>,</span></span> method</span>:</span> "</span>GET</span>"</span>,</span></span> url</span>:</span> "</span>https://mail.example.com/mail/</span>"</span>,</span></span> capture_body</span>:</span> true</span>,</span></span> checks</span>:</span> {</span></span> status_in</span>:</span> [</span>200</span>]</span>,</span></span> body_includes</span>:</span> [</span>"</span>Roundcube Webmail :: Welcome to Roundcube Webmail</span>"</span>]</span>,</span></span> }</span>,</span></span> extract</span>:</span> [</span>{</span></span> type</span>:</span> "</span>dom</span>"</span>,</span></span> selector</span>:</span> '</span>input[name="_token"]</span>'</span>,</span></span> attribute</span>:</span> "</span>value</span>"</span>,</span></span> as</span>:</span> "</span>token</span>"</span>,</span></span> }</span>]</span>,</span></span> }</span>,</span></span> {</span></span> name</span>:</span> "</span>login-submit</span>"</span>,</span></span> method</span>:</span> "</span>POST</span>"</span>,</span></span> url</span>:</span> "</span>https://mail.example.com/mail/?_task=login</span>"</span>,</span></span> headers</span>:</span> {</span> "</span>content-type</span>"</span>:</span> "</span>application/x-www-form-urlencoded</span>"</span> }</span>,</span></span> body</span>:</span> "</span>_task=login&_action=login&_timezone=...&_token={{token}}&_user=...&_pass=...</span>"</span>,</span></span> capture_body</span>:</span> true</span>,</span></span> checks</span>:</span> {</span></span> status_in</span>:</span> [</span>200</span>]</span>,</span></span> body_includes</span>:</span> [</span>"</span>Roundcube Webmail :: Inbox</span>"</span>,</span> '</span>data-label-msg="The list is empty."</span>'</span>]</span>,</span></span> }</span>,</span></span> extract</span>:</span> [</span>{</span></span> type</span>:</span> "</span>regex</span>"</span>,</span></span> pattern</span>:</span> '</span>"request_token":"([^"]+)"</span>'</span>,</span></span> group</span>:</span> 1</span>,</span></span> as</span>:</span> "</span>logout_token</span>"</span>,</span></span> }</span>]</span>,</span></span> }</span>,</span></span> {</span></span> name</span>:</span> "</span>logout</span>"</span>,</span></span> method</span>:</span> "</span>GET</span>"</span>,</span></span> url</span>:</span> "</span>https://mail.example.com/mail/?_task=logout&_token={{logout_token}}</span>"</span>,</span></span> capture_body</span>:</span> true</span>,</span></span> checks</span>:</span> {</span></span> status_in</span>:</span> [</span>200</span>]</span>,</span></span> body_includes</span>:</span> [</span>"</span>Roundcube Webmail :: Welcome to Roundcube Webmail</span>"</span>]</span>,</span></span> }</span>,</span></span> }</span>,</span></span> ]</span></span></code></pre> Each step depends on extracted values from the previous one: {{token}}</code> (the CSRF token extracted from the login form’s hidden input via a DOM CSS selector — input[name="_token"]</code>) flows into the POST body, and {{logout_token}}</code> (the request_token</code> extracted from the Roundcube JavaScript via regex) flows into the logout URL. The dom</code> extractor parses the HTML response with deno-dom’s DOMParser</code> and runs querySelector</code> — far more robust than regex for structured HTML. Session cookies are managed automatically by the native cookie_jar</code> — the roundcube_sessid</code> and roundcube_sessauth</code> cookies persist across all three steps within an iteration, and Roundcube’s logout response invalidates the session so the next iteration starts clean.</p> === per-worker results ===</span></span> </span> worker iterations login_ok logout_ok ok?</span></span> ----------------------------------------------------------------------------------</span></span> http://worker1:9091 45 45 45 PASS</span></span> http://worker2:9091 45 45 45 PASS</span></span> ----------------------------------------------------------------------------------</span></span> TOTAL 90 90 90 PASS</span></span></code></pre> All 90 iterations completed successfully across both workers: every login extracted a valid CSRF token, every POST received the inbox page, and every logout returned to the login screen. This validates the full extractor → template substitution → cookie management pipeline running inside the Rust FFI layer on remote machines.</p> Deep Dive: Performance Analysis🔗</a></h2> To understand the performance gaps, I profiled loadgen-rs using perf</code> and samply</a>. The setup:</p> #</span> Build with debug symbols but full optimization</span></span> cargo</span> build</span> -</span>-profile</span> profiling</span> #</span> inherits release, adds debug=2</span></span> </span> #</span> Record CPU profile</span></span> perf</span> record</span> -</span>g</span> -</span>-call-graph</span> dwarf</span> -</span>o</span> /tmp/perf-h2.data</span> -</span>-</span> \</span></span> ./target/profiling/loadgen-rs</span> -</span>-alpn-list</span> h2</span> -</span>D</span> 10s</span> -</span>c</span> 64</span> -</span>t</span> 4</span> -</span>m</span> 8</span> \</span></span> -</span>-insecure</span> https://127.0.0.1:8082/?s=256k</span></span></code></pre>H2 CPU Profile🔗</a></h3> Self %</th> Function</th> Category</th></tr></thead> 27%</strong></td> ring::aes_gcm_dec_update</code></td> TLS decrypt (AES-256-GCM)</td></tr> 21%</strong></td> __memmove_avx</code> (libc)</td> Memory copies</td></tr> 16%</strong></td> Kernel</td> TCP read/write syscalls</td></tr> 2.0%</td> rustls::read_opaque_message_header</code></td> TLS framing</td></tr> 1.4%</td> _mi_page_malloc</code></td> Allocator (mimalloc)</td></tr> 1.3%</td> h2::Connection::poll2</code></td> H2 protocol state machine</td></tr> 1.3%</td> h2::DynConnection::recv_frame</code></td> H2 frame processing</td></tr> 1.2%</td> BytesMut::split_to</code></td> Buffer management</td></tr> </tbody></table> ~64% of CPU time is spent outside of our code entirely</strong> — in TLS decryption, memory copies, and kernel syscalls. Only ~2.5% is in the h2 crate’s protocol logic.</p> The H2 latency gap (1.8x vs h2load) is not</em> a CPU bottleneck — it’s architectural. The h2 crate uses an async mpsc channel between SendRequest</code> (our task) and the connection driver task (h2’s internal I/O loop). Each request crosses this channel boundary at least twice (send + receive), adding idle time that doesn’t show up in CPU profiles.</p> h2load, by contrast, runs everything in a single libev event loop tick — nghttp2 processes the request, writes the frame, and reads the response without any task switching.</p> H3 CPU Profile🔗</a></h3> Self %</th> Function</th> Category</th></tr></thead> 15%</strong></td> ring::aes_gcm_dec_update</code></td> TLS decrypt</td></tr> 13%</strong></td> __memmove_avx</code> (libc)</td> Memory copies</td></tr> 12%</strong></td> Kernel</td> UDP recvmsg syscalls</td></tr> 3.5%</strong></td> quinn_proto::Connection::process_payload</code></td> QUIC packet processing</td></tr> 3.1%</strong></td> quinn_proto::Connection::handle_decode</code></td> QUIC packet decode</td></tr> 2.9%</strong></td> quinn::endpoint::RecvState::poll_socket</code></td> UDP socket I/O</td></tr> 2.8%</strong></td> binary_heap::PeekMut::pop</code></td> Timer/scheduling (loss detection, CC)</td></tr> 1.8%</td> ring::aes_gcm::open</code></td> Additional TLS overhead</td></tr> 1.7%</td> quinn::RecvStream::poll_read_chunk</code></td> Stream receive</td></tr> 1.7%</td> quinn_proto::Endpoint::handle</code></td> Packet routing/demux</td></tr> 1.6%</td> h3::proto::varint::VarInt::decode</code></td> H3 framing</td></tr> </tbody></table> The QUIC protocol stack (quinn-proto) accounts for ~15% of CPU</strong> — compared to ~2.5% for h2. This is the fundamental cost of implementing QUIC in userspace Rust:</p> Per-packet crypto</strong>: QUIC encrypts each UDP datagram individually (vs TLS record batching over TCP)</li> Congestion control state machine</strong>: Quinn maintains timer heaps (binary_heap::pop</code> at 2.8%) for loss detection and CC that TCP offloads to the kernel</li> Packet demux</strong>: Endpoint::handle</code> routes each incoming UDP datagram to the correct connection — work that the kernel’s TCP stack does for free</li> </ul> h2load uses ngtcp2 (C) and nghttp3 (C) which have tighter inner loops, less allocation overhead, and direct writev()</code> for packet batching.</p> What the Profile Means🔗</a></h3> The profiling data tells a clear story: we’ve hit the performance ceiling of the Rust crate ecosystem</strong>, not a bug in our code.</p> For H2, the bottleneck is TLS + memcpy + kernel — things we can’t optimize without replacing ring/rustls. For H3, quinn’s QUIC protocol processing adds ~15% overhead that ngtcp2 avoids by being a leaner C implementation.</p> Optimization Attempts (What Worked and What Didn’t)🔗</a></h3> During development, I tried several optimizations to close the gap with h2load:</p> What Worked🔗</a></h4> Optimization</th> Effect</th> Protocol</th></tr></thead> Raw h2 crate instead of hyper</td> Eliminated one abstraction layer</td> H2</td></tr> Raw TCP + httparse instead of hyper for H1</td> +15-25% RPS vs hyper-based H1</td> H1</td></tr> initial_rtt(1ms)</code> for quinn</td> Faster congestion control ramp-up on low-latency links</td> H3</td></tr> 1 GB stream/connection windows</td> Matches h2load’s window sizes</td> H2, H3</td></tr> H2 flow control: 512 KB batch release</td> Reduces WINDOW_UPDATE frame overhead</td> H2</td></tr> UDP socket buffers 4 MB via socket2</td> Fewer kernel drops under load</td> H3</td></tr> mimalloc global allocator</td> Better multi-threaded malloc performance</td> All</td></tr> Pre-built request templates</td> Avoid re-building HTTP requests per iteration</td> All</td></tr> Per-worker current-thread runtimes</td> No cross-thread lock contention</td> All</td></tr> </tbody></table> What Didn’t Work🔗</a></h4> Optimization</th> Expected</th> Actual</th> Why</th></tr></thead> Thread-local shared quinn::Endpoint</td> Less per-connection overhead</td> -12% RPS</strong></td> UDP socket contention — all connections on one worker competed for the same socket</td></tr> H2 immediate flow control release (per-chunk)</td> Lower latency</td> -5.6% RPS</strong></td> Generated too many WINDOW_UPDATE frames and cross-task messages</td></tr> Cubic congestion controller (instead of NewReno)</td> Match h2load’s ngtcp2 CC</td> -7% RPS</strong></td> Quinn’s Cubic implementation doesn’t scale well with 512 concurrent connections on localhost</td></tr> http2 crate (0x676e67 fork with parking_lot)</td> Faster mutex in h2 hot path</td> -2.3% RPS</strong></td> The h2 protocol logic is only 2.5% of CPU — faster locks don’t help when locks aren’t the bottleneck</td></tr> </tbody></table> The failed optimizations were valuable lessons: profiling before</em> optimizing would have saved time. The http2 fork experiment, for instance, was motivated by the assumption that h2’s internal locking was a bottleneck — but perf showed it was only 1.3% of CPU time.</p> Architectural Comparison: loadgen-rs vs h2load🔗</a></h3> Understanding why</em> the gaps exist requires comparing the architectures:</p> H2: Async Tasks vs Single Event Loop🔗</a></h4> loadgen-rs: Worker Task ──mpsc──▶ h2 Connection Driver Task ──▶ TCP</span></span> h2load: libev event loop ──▶ nghttp2 ──▶ writev() ──▶ TCP</span></span></code></pre> The h2 crate splits request handling across two async tasks connected by an mpsc channel. Each request crosses this boundary twice. h2load runs everything synchronously in one event loop tick — zero context switches per request.</p> This explains the latency gap (1.8x) without a throughput gap: the pipeline stays full, but each individual request takes longer to complete.</p> H3: Userspace QUIC vs Lean C Implementation🔗</a></h4> loadgen-rs: quinn (Rust) ──▶ ring (Rust/asm) ──▶ UDP socket</span></span> h2load: ngtcp2 (C) ──▶ OpenSSL/BoringSSL (C/asm) ──▶ UDP socket</span></span></code></pre> Both implement the same QUIC protocol, but ngtcp2 has advantages:</p> Tighter C inner loops with less allocation</li> Direct sendmsg</code>/recvmsg</code> batching</li> Fewer abstraction layers between protocol logic and I/O</li> </ul> Quinn adds Tokio task scheduling, channel-based I/O between endpoint and connection state, and Rust’s safety overhead (bounds checks, reference counting on Bytes</code>).</p> Deep Dive: Internal Architecture🔗</a></h2> Enum-Based Driver Dispatch🔗</a></h3> Instead of dyn Trait</code> with async boxing, I use an enum:</p> pub</span> enum</span> Connection</span> {</span></span> H1Raw</span>(</span>h1_raw</span>::</span>H1RawConnection</span>)</span>,</span></span> H2</span>(</span>h2</span>::</span>H2Connection</span>)</span>,</span></span> H3</span>(</span>h3</span>::</span>H3Connection</span>)</span>,</span></span> }</span></span> </span> impl</span> Connection</span> {</span></span> pub</span> async</span> fn</span> send_request</span>(</span>&</span>mut</span> self</span>,</span> config</span>:</span> &</span>RequestConfig</span>)</span></span> -></span> Result</span><</span>RequestResult</span>,</span> RequestError</span>></span></span> {</span></span> match</span> self</span> {</span></span> Connection</span>::</span>H1Raw</span>(</span>c</span>)</span> =></span> c</span>.</span>send_request</span>(</span>config</span>)</span>.</span>await</span>,</span></span> Connection</span>::</span>H2</span>(</span>c</span>)</span> =></span> c</span>.</span>send_request</span>(</span>config</span>)</span>.</span>await</span>,</span></span> Connection</span>::</span>H3</span>(</span>c</span>)</span> =></span> c</span>.</span>send_request</span>(</span>config</span>)</span>.</span>await</span>,</span></span> }</span></span> }</span></span> }</span></span></code></pre> Benefits:</p> No heap allocation per dispatch</li> Compiler can inline</li> No Pin<Box<dyn Future>></code> overhead</li> </ul> FFI Two-API Design (Batch vs Step)🔗</a></h3> The Deno integration isn’t just a thin wrapper — it’s built on a Cargo workspace with a dedicated loadgen-ffi</code> crate that compiles to a 6.4 MB shared library (libloadgen_ffi.so</code>) exposing 13 C ABI functions via Deno.dlopen()</code>.</p> Two distinct FFI APIs🔗</a></h4> The shared library provides two API levels, each serving a different use case:</p> 1. Batch API</strong> — for high-throughput load generation:</p> loadgen_create(config_json) → handle</span></span> loadgen_run(handle) → report_json (nonblocking in Deno)</span></span> loadgen_metrics_snapshot(handle) → json</span></span> loadgen_destroy(handle)</span></span></code></pre> This wraps the existing run_from_config()</code> engine — the same optimized H1/H2/H3 drivers, per-worker runtimes, and HdrHistograms that match h2load performance. A single loadgen_run</code> call spawns the full benchmark and returns a complete RunReport</code> as JSON.</p> 2. Step API</strong> — for scripted request sequences with correlation:</p> loadgen_step_session_create(session_config_json) → session_handle</span></span> loadgen_step_execute(session_handle, request_json) → response_json (nonblocking)</span></span> loadgen_step_session_reset(session_handle)</span></span> loadgen_step_session_destroy(session_handle)</span></span></code></pre> The Step API uses reqwest</strong> as its HTTP client rather than the custom drivers. This is a deliberate architectural choice: reqwest provides built-in cookie jars, redirect following, H1/H2/H3 support, and connection pooling — features that the raw benchmark drivers intentionally omit for performance. In script mode, correctness (cookies, redirects, body capture) matters more than raw throughput.</p> Panic-safe FFI boundary🔗</a></h4> Every exported function is wrapped in std::panic::catch_unwind()</code> to prevent Rust panics from unwinding across the C ABI boundary (which would be undefined behavior). Errors are stored in a thread-safe slot (OnceLock<Mutex<Option<String>>></code>) and retrieved via loadgen_last_error()</code>. All returned char</code> strings must be freed with loadgen_free_string()</code> — the contract is documented in docs/ffi-contract.md</code>.</p> TypeScript script engine🔗</a></h4> On the Deno side, runScriptScenario()</code> in ts/script_mode.ts</code> implements a k6-inspired scenario runner with:</p> VU parallelism</strong>: N virtual users run as concurrent Promise.all</code> tasks</li> Duration-based execution</strong>: VUs loop until the target duration expires</li> Two execution modes</strong>: fetch</code> (Deno’s native HTTP client) and ffi-step</code> (Rust reqwest via the Step API)</li> Template engine</strong>: {{variable}}</code> substitution in URLs, headers, and bodies</li> Extractor system</strong>: Capture values from JSON paths, response headers, regex groups, or DOM CSS selectors (via deno-dom) — stored in per-VU state for use in subsequent steps</li> Check system</strong>: Eight check types including status_in</code>, body_includes</code>, header_exists</code>, header_equals</code>, header_includes</code>, json_path_exists</code>, json_path_equals</code>, and regex_match</code></li> Cookie jar</strong>: Full RFC-aware implementation in TypeScript (for fetch</code> mode) with domain/path matching, expiry handling, and Secure attribute support</li> </ul> The ffi-step</code> mode delegates cookie management and redirects to reqwest on the Rust side, while the fetch</code> mode handles everything in TypeScript. Both modes share the same check and extractor evaluation logic.</p> The full scripting documentation is in loadgen-script</a></p> Example coverage🔗</a></h4> The examples/</code> directory in loadgen-rs repository</a> demonstrates the full feature surface:</p> Example</th> What it demonstrates</th></tr></thead> simple.ts</code></td> Minimal FFI benchmark (10 requests, single client)</td></tr> correlation_mvp.ts</code></td> Login, extract JWT token, use in follow-up request</td></tr> cookie_redirect_local_mvp.ts</code></td> Embedded Deno server with Set-Cookie + 302 redirect chain</td></tr> protocol-matrix.ts</code></td> H1/H2/H3 batch + script mode for all three protocols</td></tr> ab-stages.ts</code></td> Ramping VUs with stage interpolation (k6-like stages)</td></tr> native_step_skeleton.ts</code></td> Direct Step API usage without the scenario runner</td></tr> all-functionality.ts</code></td> Full integration test suite using Chai assertions</td></tr> </tbody></table> The cookie_redirect_local_mvp.ts</code> example is particularly interesting: it spins up a local Deno HTTP server that sets cookies and issues redirects, then runs the script scenario against it — a self-contained integration test that validates the entire cookie + redirect + check pipeline.</p> Appendix: Commands and Reproducibility🔗</a></h2> Equivalent Command Pairs (h2load vs loadgen-rs)🔗</a></h3> To compare both tools fairly, use equivalent flags and remember: --rps</code> is per client</strong> in both tools.</p> #</span> 1) H1 count mode (same request budget)</span></span> h2load</span> -</span>-h1</span> -</span>n</span> 200</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-h1</span> -</span>n</span> 200</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients 20 rps/client)</span></span> h2load</span> -</span>-h1</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-h1</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 3) H2 duration mode with pacing (same effective target)</span></span> h2load</span> -</span>-alpn-list=h2</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-alpn-list</span> h2</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> </span> #</span> 4) H3 duration mode with pacing (same effective target)</span></span> h2load</span> -</span>-alpn-list=h3</span> -</span>D</span> 10s</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span> loadgen-rs</span> -</span>-alpn-list</span> h3</span> -</span>-duration</span> 10s</span> -</span>n</span> 0</span> -</span>c</span> 10</span> -</span>t</span> 8</span> -</span>m</span> 1</span> -</span>-rps</span> 20</span> "</span>https://bench.local:8082/?s=256k</span>"</span></span></code></pre>CLI Compatibility Map🔗</a></h3> The table below maps loadgen-rs</code> options to their closest h2load</code> equivalent.</p> loadgen-rs</code></th> h2load</code></th> Mapping</th> Notes</th></tr></thead> <URL></code></td> <URI>...</code></td> partial</td> h2load</code> accepts multiple URIs</td></tr> -n <REQUESTS></code></td> -n, --requests=<N></code></td> same</td> </td></tr> -D, --duration</code></td> -D, --duration</code></td> same</td> </td></tr> --warm-up-time</code></td> --warm-up-time</code></td> same</td> </td></tr> --ramp-up-time</code></td> none</code></td> no direct equivalent</td> Closest knobs are -r/--rate</code> + --rate-period</code>, but semantics differ</td></tr> -c <CLIENTS></code></td> -c, --clients=<N></code></td> same</td> </td></tr> -t <THREADS></code></td> -t, --threads=<N></code></td> same</td> </td></tr> -m <MAX_STREAMS></code></td> -m, --max-concurrent-streams=<N></code></td> same</td> </td></tr> --h1</code></td> --h1</code></td> same</td> </td></tr> --alpn-list h2\|h3</code></td> --alpn-list=<LIST></code></td> partial</td> h2load</code> allows arbitrary ALPN lists</td></tr> --connect-timeout</code></td> none</code></td> no direct equivalent</td> Only related connection-level timeout controls exist (-T</code>, -N</code>)</td></tr> --request-timeout</code></td> none</code></td> no direct equivalent</td> No per-request timeout flag in h2load</code></td></tr> --tcp-quickack</code></td> none</code></td> none</td> </td></tr> --method <METHOD></code></td> none</code></td> no direct equivalent</td> h2load</code> is primarily GET; POST</code> is enabled via -d</code></td></tr> -H, --header</code></td> -H, --header</code></td> same</td> </td></tr> -d, --data <DATA></code></td> -d, --data=<PATH></code></td> partial</td> loadgen-rs</code>: inline body; h2load</code>: file path</td></tr> --data-file <PATH></code></td> -d, --data=<PATH></code></td> close equivalent</td> </td></tr> --rps <RPS></code></td> --rps=<N></code></td> same</td> per-client target in both tools (aggregate RPS * clients</code>)</td></tr> -k, --insecure</code></td> none</code></td> no CLI equivalent</td> No direct insecure toggle in h2load</code> CLI</td></tr> --tls-ciphers</code></td> --ciphers</code>, --tls13-ciphers</code></td> partial</td> h2load</code> splits TLS <=1.2 and TLS 1.3 ciphers</td></tr> --tls-ca <PATH></code></td> none</code></td> none</td> </td></tr> -o, --output <PATH></code></td> none</code></td> none</td> h2load</code> offers --log-file</code> (per-request TSV), not run-level JSON/CSV</td></tr> --format jsonl\|csv</code></td> none</code></td> none</td> </td></tr> --no-human</code></td> none</code></td> none</td> </td></tr> --tail-friendly</code></td> none</code></td> none</td> </td></tr> --metrics-sample</code></td> none</code></td> none</td> </td></tr> </tbody></table> Container Baseline🔗</a></h3> The multi-stage Containerfile builds the binary in a Rust image and copies it into a minimal Debian Slim image with bash</code> and curl</code>, with HTTP/3 support built in:</p> podman</span> build</span> -</span>t</span> loadgen-rs</span> -</span>f</span> Containerfile</span> .</span></span> podman</span> run</span> -</span>-rm</span> -</span>-network</span> host</span> \</span></span> -</span>-cap-add=SYS_ADMIN</span> \</span></span> -</span>-security-opt</span> seccomp=unconfined</span> \</span></span> loadgen-rs</span> -</span>D</span> 10s</span> -</span>c</span> 8</span> -</span>t</span> 4</span> -</span>-alpn-list=h2</span> https://target:8443/</span></span></code></pre> --network host</code> avoids NAT overhead. --cap-add=SYS_ADMIN</code> enables io_uring inside the container.</p> Test Server Setup🔗</a></h3> The test server used in this post is haterm</code> from HAProxy</code>, created and started with the following commands:</p> # create combined.pem in TLS-PATH</span></span> cat server.crt ca.crt server.key > combined.pem</span></span> </span> # build container image</span></span> buildah bud -f Containerfile-hap-git -t hap-own</span></span> </span> # run haterm</span></span> podman run -d --rm --name hap-own \</span></span> --network host \</span></span> --entrypoint=/usr/local/sbin/haterm \</span></span> -v <TLS-PATH>/tls:/mnt:ro \</span></span> localhost/haterm:latest \</span></span> -L 127.0.0.1:8081 \</span></span> -F 'bind quic4@127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h3' \</span></span> -F 'bind 127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h2'</span></span></code></pre> The used Containerfile-hap-git</a></p> Cloud Deployment (Terraform + Ansible)🔗</a></h3> Running distributed benchmarks on your local machines is useful for development, but real load testing demands dedicated cloud infrastructure — consistent hardware, dedicated NICs, and geographic proximity to the target. loadgen-rs ships with a complete Terraform + Ansible pipeline for Hetzner Cloud that takes you from zero to running workers in under five minutes.</p> The Terraform configuration provisions dedicated-vCPU instances (CCX series), places them in a private network, and configures a firewall that allows SSH and the worker-agent port. A single generate-inventory.sh</code> script bridges the gap between Terraform and Ansible — it reads the Terraform outputs and writes a ready-to-use ansible/inventory.ini</code> with the public IPs:</p> terraform apply</span></span> │</span></span> ├── hcloud_server.worker[0] → 49.13.x.x (loadgen-worker-0)</span></span> ├── hcloud_server.worker[1] → 49.13.y.y (loadgen-worker-1)</span></span> ├── hcloud_network → 10.0.1.0/24 (private)</span></span> └── hcloud_firewall → SSH + port 9091</span></span> │</span></span> ▼</span></span> ./generate-inventory.sh</span></span> │</span></span> ▼</span></span> ansible/inventory.ini</span></span> ├── loadgen-worker-0 ansible_host=49.13.x.x worker_port=9091</span></span> └── loadgen-worker-1 ansible_host=49.13.y.y worker_port=9091</span></span> │</span></span> ▼</span></span> ansible-playbook deploy-workers.yml</span></span> │</span></span> ▼</span></span> Workers ready — POST /run from controller</span></span></code></pre> The full workflow from provisioning to results:</p> #</span> 1. Provision Hetzner Cloud machines</span></span> cd</span> terraform</span></span> cp</span> terraform.tfvars.example</span> terraform.tfvars</span></span> #</span> Set: hcloud_token, ssh_key_name, worker_count=4, worker_type=ccx33</span></span> terraform</span> init</span> &&</span> terraform</span> apply</span></span> </span> #</span> 2. Generate Ansible inventory from Terraform state</span></span> ./generate-inventory.sh</span></span> #</span> → ansible/inventory.ini with 4 workers</span></span> </span> #</span> 3. Deploy + benchmark + cleanup in one command</span></span> cd</span> ..</span></span> scripts/distributed-remote.sh</span> \</span></span> -</span>i</span> ansible/inventory.ini</span> \</span></span> -</span>-target-url</span> "</span>https://bench.target:8443/?s=256k</span>"</span> \</span></span> -</span>c</span> 1000</span> -</span>D</span> 60</span> -</span>-stop-after</span></span> </span> #</span> 4. Tear down when done</span></span> cd</span> terraform</span> &&</span> terraform</span> destroy</span></span></code></pre> The Ansible playbook (deploy-workers.yml</code>) handles everything on the remote machines: installing Deno if not present, copying the FFI shared library and TypeScript files, deploying a systemd service unit (with a tmux fallback for systems without systemd), and running health checks to confirm each worker is ready. The matching stop-workers.yml</code> playbook cleanly shuts everything down.</p> This separation of concerns — Terraform owns infrastructure, Ansible owns configuration, the shell script orchestrates the workflow — means you can mix and match. Already have machines? Skip Terraform, write inventory.ini</code> by hand. Prefer a different cloud? Replace the .tf</code> files, keep the same Ansible playbooks and scripts. The worker-agent doesn’t care how it got deployed.</p> Component</th> Responsibility</th></tr></thead> terraform/.tf</code></td> Provision Hetzner VMs, network, firewall</td></tr> terraform/generate-inventory.sh</code></td> Bridge: Terraform outputs → Ansible inventory</td></tr> ansible/deploy-workers.yml</code></td> Install Deno, deploy files, start workers</td></tr> ansible/stop-workers.yml</code></td> Stop workers (systemd or tmux)</td></tr> scripts/distributed-remote.sh</code></td> End-to-end: deploy → benchmark → stop</td></tr> </tbody></table> Conclusion🔗</a></h2> loadgen-rs is a lean, focused tool: it performs exactly one run, outputs the results as JSONL, and leaves the orchestration (warmup, parameter sweeps, plotting) to the calling script. The combination of raw h2, quinn, and rustls covers all three HTTP protocols — and thanks to per-worker runtimes and a lock-free design, the client itself is not the bottleneck.</p> CPU profiling confirmed this: over 60% of execution time is spent in TLS decryption, memory copies, and kernel syscalls — code paths shared by any HTTP benchmark client regardless of language. The remaining gaps to h2load are architectural properties of the Rust crate ecosystem (h2’s async channel design, quinn’s userspace QUIC overhead) rather than optimization opportunities in loadgen-rs itself.</p> With the Deno FFI layer, distributed mode, and Terraform/Ansible pipeline, loadgen-rs now serves five usage patterns: shell scripts that need a fast, h2load-compatible CLI tool; TypeScript users who want k6-style scripted scenarios with native H1/H2/H3 performance; multi-machine deployments where a single process can’t generate enough load; cloud-provisioned benchmarks where you spin up dedicated Hetzner instances, run the test, and tear everything down in minutes; and distributed scripted scenarios where workers execute multi-step request sequences with checks and response validation on remote machines. The two-API design (batch for throughput, step for correlation) plus the controller/worker/infrastructure stack bridges all five without compromising any of them.</p> Code and Repository Link🔗</a></h3> The code is on loadgen-rs - GitHub</a> — PRs welcome.</p> cert-manager-webhook-libdns: One Webhook, Many DNS Providers 2026-03-03T00:00:00+00:00 cert-manager-webhook-libdns: One Webhook, Many DNS Providers🔗</a></h1> Managing DNS-01 challenges with cert-manager often means choosing one provider-specific integration and staying tightly coupled to it. cert-manager-webhook-libdns</a> takes a different approach: it uses the libdns</a> ecosystem so one webhook can support many DNS providers through a shared interface.</p> If you run mixed infrastructure, migrate between DNS vendors, or just want a cleaner operational model, this project is built for that.</p> Why this project exists🔗</a></h2> In many Kubernetes setups, certificate automation becomes harder than it should be because DNS integrations are fragmented:</p> Every provider has different APIs, credentials, and operational quirks.</li> Switching providers often means switching tooling.</li> Testing and maintenance can become provider-specific and brittle.</li> </ul> cert-manager-webhook-libdns</a> solves this by exposing a single cert-manager webhook that delegates DNS record operations to pluggable libdns providers.</p> What it does🔗</a></h2> At a high level, the webhook:</p> Receives DNS-01 challenge requests from cert-manager.</li> Loads provider credentials from Kubernetes Secrets.</li> Resolves and writes challenge TXT records through the selected libdns provider.</li> Cleans up records after validation.</li> </ul> It supports important real-world behaviors, including:</p> Handling multiple TXT values for wildcard + apex certificate requests.</li> Safer fallback logic when record listing fails.</li> Provider-specific TTL handling (for example, deSEC minimum TTL enforcement).</li> Deterministic provider listing in CLI output.</li> </ul> Operational improvements in recent iterations🔗</a></h2> Several practical improvements were added to make this project easier to run and maintain:</p> Semantic version output via --version</code> (currently v0.5.0</code>).</li> No etcd/envtest dependency in the default make test</code> path.</li> Cleaner container workflow with Containerfile</code> support.</li> Release workflow hardening:</li> tests run before image release</li> latest</code> is only published for stable tags</li> release creation happens after successful image push</li> concurrency guard to prevent overlapping tag releases</li> </ul> These are the kinds of changes that reduce surprises in CI/CD and day-to-day operations.</p> Provider compatibility is now automated🔗</a></h2> A major maintenance challenge with libdns-based projects is knowing which upstream provider modules are actually compatible with your pinned libdns</code> version.</p> This repository now includes an automated compatibility process:</p> Script: scripts/check_libdns_provider_compat.sh</code></li> Workflow: .github/workflows/libdns-provider-compat.yml</code></li> Report snapshot: docs/01_provider-compat-latest.md</code></li> Historical snapshots: docs/YYYY-MM-DD_provider-compat.md</code></li> </ul> The check uses gh api</code> to scan provider repositories, inspect their go.mod</code>, and classify compatibility against the libdns version used in this project.</p> That gives maintainers a repeatable compatibility signal instead of manual guesswork.</p> Quick start🔗</a></h2> 1. Build and publish the image🔗</a></h3> Use your preferred container runtime (Podman is preferred, Docker fallback is supported):</p> make</span> container-build</span></span> make</span> container-push</span> IMAGE_TAG=v0.5.0</span> REGISTRY=ghcr.io/</span><</span>your-or</span>g</span>></span></span></code></pre>2. Deploy the Helm chart🔗</a></h3> helm</span> install</span> libdns-webhook</span> ./deploy/libdns-webhook</span> \</span></span> -</span>-namespace</span> cert-manager</span> \</span></span> -</span>-set</span> image.repository=ghcr.io/</span><</span>your-or</span>g</span>></span>/cert-manager-webhook-libdns</span> \</span></span> -</span>-set</span> image.tag=v0.5.0</span> \</span></span> -</span>-set</span> groupName=acme.yourdomain.com</span></span></code></pre>3. Create provider credentials secret🔗</a></h3> Example for token-based providers:</p> a</span>piVersion</span>:</span> v</span>1</span></span> k</span>ind</span>:</span> S</span>ecret</span></span> m</span>etadata</span>:</span></span> n</span>ame</span>:</span> d</span>ns-provider-credentials</span></span> n</span>amespace</span>:</span> c</span>ert-manager</span></span> t</span>ype</span>:</span> O</span>paque</span></span> s</span>tringData</span>:</span></span> a</span>pi_token</span>:</span> "</span><YOUR_API_TOKEN></span>"</span></span></code></pre>4. Configure ClusterIssuer🔗</a></h3> a</span>piVersion</span>:</span> c</span>ert-manager.io/v1</span></span> k</span>ind</span>:</span> C</span>lusterIssuer</span></span> m</span>etadata</span>:</span></span> n</span>ame</span>:</span> l</span>etsencrypt-prod</span></span> s</span>pec</span>:</span></span> a</span>cme</span>:</span></span> s</span>erver</span>:</span> h</span>ttps://acme-v02.api.letsencrypt.org/directory</span></span> e</span>mail</span>:</span> y</span>our-email@example.com</span></span> p</span>rivateKeySecretRef</span>:</span></span> n</span>ame</span>:</span> l</span>etsencrypt-prod-account</span></span> s</span>olvers</span>:</span></span> -</span> d</span>ns01</span>:</span></span> w</span>ebhook</span>:</span></span> g</span>roupName</span>:</span> a</span>cme.yourdomain.com</span></span> s</span>olverName</span>:</span> l</span>ibdns</span></span> c</span>onfig</span>:</span></span> p</span>rovider</span>:</span> c</span>loudflare</span></span> s</span>ecretRef</span>:</span></span> n</span>ame</span>:</span> d</span>ns-provider-credentials</span></span> n</span>amespace</span>:</span> c</span>ert-manager</span></span></code></pre>Useful CLI checks🔗</a></h2> Before deployment, you can quickly inspect the binary:</p> ./webhook</span> -</span>-version</span></span> ./webhook</span> -</span>-list-providers</span></span></code></pre> This is helpful for validating exactly what was compiled into your image.</p> Who should use this🔗</a></h2> This project is a strong fit if you:</p> Run cert-manager and use DNS-01 challenges.</li> Need flexibility across multiple DNS providers.</li> Want one consistent webhook integration instead of many provider-specific implementations.</li> Care about release discipline and compatibility visibility.</li> </ul> Final thoughts🔗</a></h2> cert-manager-webhook-libdns</a> is not just a DNS integration layer. It is an operational strategy: keep cert-manager integration stable while letting providers vary underneath.</p> If your platform team wants fewer moving parts and more predictable certificate automation, this is a practical approach worth adopting.</p> Simple autotls setup with ferron 2025-10-22T00:00:00+00:00 Introduction🔗</a></h1> ferron</a> is a webserver written in rust. This webserver have the Automatic TLS</a> feature for deSEC</a> implemented which makes it quite easy to create a TLS http server listen on localhost 🤩 🥳.</p> Pre-Requirements:</p> curl</a></li> deSEC Account</a></li> own Domain => referred in this post as DOMAIN</code></li> </ul> Get Token.🔗</a></h1> To be able to add the acme challenge to the deSEC is a Token required. How to get and mange tokens for deSEC is documented at Manage Tokens</a></p> config and start ferron🔗</a></h1> The config file ferron.kdl</code>.</p> {</span></span> </span> protocols "h1" "h2" "h3"</span></span> </span> log "/dev/stdout"</span></span> error_log "/dev/stderr"</span></span> </span> default_http_port 8080 // for real webserver change the port to 80</span></span> default_https_port 8443 // for real webserver change the port to 443</span></span> </span> trust_x_forwarded_for #true</span></span> </span> auto_tls</span></span> auto_tls_contact "Your-letsencrypt@Email-address"</span></span> auto_tls_cache "letsencrypt-cache"</span></span> auto_tls_letsencrypt_production</span></span> auto_tls_challenge "dns-01" provider="desec" api_token="THE-TOKEN"</span></span> </span> }</span></span> </span> .DOMAIN {</span></span> </span></span> status 200 url="/static_return" body="OK"</span></span> </span> root "wwwroot"</span></span> }</span></span></code></pre> After the Install</a> process can you use the above config file to start ferron.</p> # ./ferron -c /datadisk/Downloads/ferronweb/ferron-2.0.0-beta.16/ferron.kdl</span></span> HTTP server is listening on [::]:8080...</span></span> HTTPS server is listening on [::]:8443...</span></span> HTTP/3 server is listening on [::]:8443...</span></span></code></pre> tip </div> tip</strong></span> </div> Wait at least ~2 minutes before the first request because the deSEC API and DNS Server needs some time to propagate the acme challenge.</p> </div> </div> Finally🔗</a></h1> Wenn everything works as designed should can you cal curl on localhost with an valid TLS Certificate.</p> curl -v --compressed \</span></span> --resolve test2.DOMAIN:8443:127.0.0.1 \</span></span> --http2 --http2-prior-knowledge \</span></span> -H 'X-Forwarded-For: 10.1.1.1,20.2.2.2' \</span></span> https://test2.DOMAIN:8443/static_return</span></span> </span></code></pre> ferron shows then the right IP from the first X-Forwarded-For</code> entry.</p> # ./ferron -c /datadisk/Downloads/ferronweb/ferron-2.0.0-beta.16/ferron.kdl</span></span> HTTP server is listening on [::]:8080...</span></span> HTTPS server is listening on [::]:8443...</span></span> HTTP/3 server is listening on [::]:8443...</span></span> 10.1.1.1 - - [22/Oct/2025:23:08:02 +0200] "GET /static_return HTTP/2.0" 200 2 "-" "curl/8.5.0"</span></span></code></pre> How to automate TLSA update 2025-09-28T00:00:00+00:00 Introduction🔗</a></h1> “DNS-based Authentication of Named Entities (DANE)” is specified in rfc6698</a> and it requires DNSSEC (DANE only works when DNSSEC is enabled). DNSSEC is defined in RFC4033</a>, RFC4034</a>, and RFC4035</a>.</p> I describe here my process how I updated auotmatically the TLSA record for my mailserver via the deSEC rrsets API</a>. If you like please be so kind and Donate</a> to deSEC</p> Pre-Requirements:</p> jq</a></li> curl</a></li> certbot</a></li> deSEC Account</a></li> own Domain</li> </ul> DeSEC, domain + mail setup🔗</a></h1> The “Mail setup for deSEC” is described at the link Mail setup for deSEC</a>.</p> certbot🔗</a></h1> Certbot offers the option to run some hooks at certificate handling Pre and Post Validation Hooks</a>.</p> json Payload🔗</a></h2> The file tlsa-update.json</code> have this content.</p> [</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>_25._tcp.mail</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TLSA</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>3 1 1 ${TLSA_SHA}</span>"</span></span> ]</span></span> }</span></span> ]</span></span></code></pre>post-hook🔗</a></h2> At the end add the following script into the post hook directory /etc/letsencrypt/renewal-hooks/post/</code>.</p> I called it reload-services.sh</code></p> #!</span>/bin/bash</span></span> </span> #</span># Update TLSA record</span></span> </span> export</span> DESEC_TOKEN</span>=</span><</span>YOUR_DESEC_TOKEN</span>></span></span> export</span> DOMAIN_NAME</span>=</span><</span>YOUR_DOMAIN</span>></span></span> export</span> CERT_DOMAIN</span>=</span><</span>CERT_DOMAIN</span>></span>.</span>$</span>{</span>DOMAIN_NAME</span>}</span></span> export</span> TLSA_SHA</span>=</span>$(</span>/usr/bin/openssl</span> x509</span> -</span>in</span> /etc/letsencrypt/live/</span>$</span>{</span>DOMAIN</span>}</span>/fullchain.pem</span> -</span>noout</span> -</span>pubkey</span> \</span></span> \|</span> /usr/bin/openssl</span> pkey</span> -</span>pubin</span> -</span>outform</span> DER</span> \</span></span> \|</span> /usr/bin/openssl</span> sha256</span> 2</span>>&1</span>\</span></span> \|</span> /usr/bin/cut</span> -</span>f2</span> -</span>d</span> '</span> '</span>)</span></span> </span> envsubst</span> <</span> tlsa-update.json</span> \</span></span> \|</span> curl</span> -</span>sSLX</span> PUT</span> https://desec.io/api/v1/domains/</span>$</span>{</span>DOMAIN_NAME</span>}</span>/rrsets/</span> \</span></span> -</span>-header</span> "</span>Authorization: Token </span>$</span>{</span>DESEC_TOKEN</span>}</span>"</span> \</span></span> -</span>-header</span> "</span>Content-Type: application/json</span>"</span> \</span></span> -</span>-data</span> @-</span> \</span></span> \|</span> jq</span></span> </span> /usr/bin/systemctl</span> restart</span> nginx.service</span></span> /usr/bin/systemctl</span> restart</span> dovecot.service</span></span> /usr/bin/systemctl</span> restart</span> postfix.service</span></span></code></pre> I know that there is the variable CERTBOT_DOMAIN</code> and it could also be used in the script above, if it fit’s for your use-case</p> Finally🔗</a></h1> Wenn everything works as designed should you never need to update the TLSA record when the LE certificate was changed.</p> How to Run Tor Arti as an HTTP Proxy or Behind a g3 HTTP Proxy 2025-09-01T00:00:00+00:00 Introduction🔗</a></h1> This is a short guide on how to use Tor</a> through an HTTP proxy.</p> The setup uses three components:</p> arti</a></li> g3proxy</a></li> Any software that supports an HTTP proxy</li> </ul> The diagram below shows the overall flow. --- config: look: handDrawn theme: neutral --- flowchart LR A[Client] -->\|HTTP-protocol\| B[g3] -->\|SOCKS-protocol\| C[arti] --> D[TOR-Network] --> G[Original Destination] E[Client] -->\|HTTP-protocol\| B[g3] F[Client] -->\|HTTP-protocol\| C[arti] D[TOR-Network] --> H[Original Destination] D[TOR-Network] --> I[Original Destination] J[Client] -->\|SOCKS-protocol\| C[arti] </pre> </p> Arti🔗</a></h2> Quote from the Arti homepage</a>:</p> Arti is a complete rewrite of the C Tor codebase, and it is currently under active development. It is written in Rust, and it is designed to be modular, reusable, and easy to audit.</p> </blockquote> You need to build Arti before you can run it 😄. I am using an Ubuntu-based distribution, so this guide uses apt</code>.</p> This is a short summary of the original Compiling Arti</a> guide.</p> #</span> clone the repo</span></span> git</span> clone</span> https://gitlab.torproject.org/tpo/core/arti.git</span></span> </span> #</span> navigate to the directory</span></span> cd</span> arti</span></span> </span> #</span># install some necessary packages </span></span> sudo</span> apt</span> install</span> libsqlite3-dev</span></span> </span> cargo</span> build</span> -</span>p</span> arti</span> -</span>-release</span></span></code></pre> tip </div> tip</strong></span> </div> You may need to run the build/install process more than once while adding missing dependencies on your build host.</p> </div> </div> g3proxy🔗</a></h2> You can either download</a> prebuilt g3proxy packages or build it yourself. I chose to build it.</p> git</span> clone</span> https://github.com/bytedance/g3</span></span> cd</span> g3</span></span> </span> sudo</span> apt</span> install</span> liblua5.4-dev</span> capnproto</span> libc-ares-dev</span></span> </span> cargo</span> build</span></span></code></pre> tip </div> tip</strong></span> </div> You may need to run the build/install process more than once while adding missing dependencies on your build host.</p> </div> </div> g3proxy has many more configuration options than shown here. See the project README</a>.</p> Config🔗</a></h3> g3proxy requires a configuration file.</p> ---</span></span> r</span>untime</span>:</span></span> t</span>hread_number</span>:</span> 2</span></span> </span> l</span>og</span>:</span> s</span>tdout</span></span> </span> s</span>erver</span>:</span></span> -</span> n</span>ame</span>:</span> h</span>ttp_proxy</span></span> e</span>scaper</span>:</span> d</span>efault</span></span> t</span>ype</span>:</span> h</span>ttp_proxy</span></span> l</span>isten</span>:</span></span> a</span>ddress</span>:</span> "</span>[::]:10087</span>"</span> #</span> <<<< http-proxy listening port</span></span> t</span>ls_client</span>:</span> {</span> }</span> #</span> Open layer-7 https forward forwarding support</span></span> </span> e</span>scaper</span>:</span></span> -</span> n</span>ame</span>:</span> d</span>efault</span></span> t</span>ype</span>:</span> p</span>roxy_socks5</span></span> p</span>roxy_addr</span>:</span> "</span>[::1]:9150</span>"</span> #</span> <<<< arti listening port</span></span></code></pre>Runtime🔗</a></h2> After both tools (Arti and g3proxy) are built, open two shells: run Arti in one shell and g3proxy in the other.</p> Shell 1: Arti🔗</a></h3> /datadisk/git-repos/arti</span> $</span></span> #</span> target/release/arti proxy</span></span> 2025-09-01T15:11:17Z</span> INFO</span> arti::subcommands::proxy:</span> Starting</span> Arti</span> 1.5.0</span> in</span> SOCKS</span> proxy</span> mode</span> on</span> localhost</span> port</span> 9150</span> ...</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_memquota::mtracker:</span> Memory</span> quota</span> tracking</span> initialised</span> max=</span>8.00</span> GiB</span> low_water=</span>6.00</span> GiB</span></span> 2025-09-01T15:11:17Z</span> INFO</span> arti_client::client:</span> Using</span> keystore</span> from</span> "</span>/home/alex/.local/share/arti/keystore</span>"</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_dirmgr:</span> Marked</span> consensus</span> usable.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_dirmgr:</span> Loaded</span> a</span> good</span> directory</span> from</span> cache.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> arti::subcommands::proxy:</span> Sufficiently</span> bootstrapped</span>;</span> system</span> SOCKS</span> now</span> functional.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> arti::socks:</span> Listening</span> on</span> [::1</span>]:9150.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> arti::socks:</span> Listening</span> on</span> 127.0.0.1:9150.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_dirmgr:</span> Marked</span> consensus</span> usable.</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_dirmgr:</span> Directory</span> is</span> complete.</span> attempt=</span>1</span></span> 2025-09-01T15:11:17Z</span> INFO</span> tor_guardmgr::guard:</span> We</span> have</span> found</span> that</span> guard</span> [scrubbed</span>] is usable.</span></span> 2025-09-01T15:11:18Z</span> INFO</span> arti::reload_cfg:</span> Successfully</span> reloaded</span> configuration.</span></span> 2025-09-01T15:11:28Z</span> INFO</span> tor_guardmgr::guard:</span> We</span> have</span> found</span> that</span> guard</span> [scrubbed</span>] is usable.</span></span></code></pre>Shell 2: g3proxy🔗</a></h3> /datadisk/git-repos/g3 $</span></span> # target/debug/g3proxy --config-file g3proxy/examples/chain_socks_proxy/g3proxy.yaml</span></span></code></pre> Now configure your browser to use the HTTP proxy at 127.0.0.1:10087</code>, then test with curl</code>.</p> --- config: theme: neutral --- flowchart LR A[`curl`] -->\|`127.0.0.1:10087`\| B[g3] -->\|`SOCKS`\| C[arti] --> D[TOR-Network] -->\|`https`\| G[https://check.torproject.org/] </pre> Shell 3🔗</a></h3> #</span> curl -sSL \</span></span> --proxy</span> 127.0.0.1:10087</span> \</span></span> https://check.torproject.org/</span> \</span></span> \|</span>egrep</span> -</span>i</span> cong</span></span> </span> Congratulations.</span> This</span> browser</span> is</span> configured</span> to</span> use</span> Tor.</span></span> Congratulations.</span> This</span> browser</span> is</span> configured</span> to</span> use</span> Tor.</span></span></code></pre>Some Thoughts🔗</a></h2> Because curl</code> also supports SOCKS, you can point curl</code> directly to Arti. However, many applications support only HTTP proxying and not SOCKS. For those applications, this setup is useful. There are also other tools for HTTP(S)-to-SOCKS translation, for example Privoxy forward-socks5</code></a>. I chose g3 because it is relatively small and written in Rust 😄.</p> Update🔗</a></h2> Since Arti 1.7.0 (October 30, 2025)</a>, HTTP CONNECT</code> is supported. That means Arti can be used as an HTTP proxy endpoint, not only as a SOCKS5 endpoint.</p> How to Host a Zola-Generated Site on GitLab Pages 2025-08-26T00:00:00+00:00 Introduction🔗</a></h1> After some time without a personal blog, I decided to write down what I have learned and used. This site is mainly a personal collection of tasks, tools, and setups that I use regularly.</p> Over the years, I tried plain text, plain HTML, and several CMSes and frameworks such as WordPress, Django, Gatsby, and Hugo. For some company websites, I still use a CMS because many business users are not comfortable with Git and text editors, and a CMS offers a more user-friendly interface.</p> For my personal site, I chose Zola</a> because I wanted a Rust-based static site generator</strong>. When I checked Jamstack Site Generators</a>, I found a few Rust options 🦀 and picked Zola.</p> Why GitLab Pages?🔗</a></h2> These days, hosting a static site for free is straightforward. Many version control systems</strong> offer a Pages feature, including GitLab Pages</a>. Why GitLab and not GitHub, Codeberg, or something else? Simply put: I already had a GitLab account and I like the platform.</p> What do I publish?🔗</a></h2> After choosing the tooling, the next question was content. There are already many blogs out there 🤭 on almost every topic, and more people now use AI to get answers quickly.</p> My honest answer is: “for my own laziness” 🤓.</p> I had knowledge spread across multiple places, so I kept searching for where I had documented a particular command or setup. This blog is my way to keep that knowledge in one place that I can access from anywhere.</p> Summary🔗</a></h2> So far, the setup looks like this:</p> Tool: zola</li> Hosting: GitLab Pages</li> Content: Notes for myself, and hopefully useful for others too</li> </ul> Let’s start writing🔗</a></h2> zola🔗</a></h3> The Zola docs begin with an overview</a> and the core commands:</p> zola init none-blog</span></span> cd none-blog</span></span> git remote add origin https://gitlab.com/aleks001/none-blog.git</span></span> # original theme (until March 2026)</span></span> # git submodule add https://codeberg.org/salif/linkita.git themes/linkita</span></span> # theme used March-July 2026</span></span> # git submodule add https://github.com/thomasweitzel/zolarwind.git themes/zolarwind</span></span> # current theme (since July 2026)</span></span> git submodule add --branch v4 https://codeberg.org/salif/linkita.git themes/linkita</span></span></code></pre> The last command adds the Linkita</strong></a> theme, which is what this site currently runs on.</p> Update (March 2026): I moved from Linkita to Zolarwind</a> because I preferred the design. The commented command above reflects my original setup when I first published this post.</p> Update (July 2026): I moved back to Linkita</a>. I started publishing some posts in German alongside English, and Zolarwind was never built with multilingual support in mind — it hardcodes the site’s default language in several templates. Linkita has native per-page language handling, a working language switcher, and per-language taxonomies built in, so this time the switch was about capability, not just visual preference.</p> After that, I could start writing. You can find my blog posts here</a>, and the list will keep growing.</p> The site config is here</a>.</p> Create Content🔗</a></h3> Based on what I learned from analytics tools, I always include front matter like this:</p> ---</span></span> title: How to host a Zola-generated site on GitLab Pages</span></span> description: How to host a Zola-generated site on GitLab Pages</span></span> updated: 2025-08-26</span></span> date: 2025-08-26</span></span> taxonomies:</span></span> categories:</span></span> - zola</span></span> - gitlab</span></span> tags:</span></span> - zola</span></span> - gitlab</span></span> - rust</span></span> ---</span></span></code></pre> Then I just create Markdown files and write.</p> GitLab Pages🔗</a></h3> To use GitLab Pages</a>, you need a GitLab account and a repository. This is mine for this blog: https://gitlab.com/aleks001/none-blog/</a>.</p> Zola provides a GitLab Pages deployment guide</a>. I adapted my .gitlab-ci.yml</code> based on that guide. I also added a compression step so content is served compressed to browsers, which saves bandwidth.</p> This is my .gitlab-ci.yml</code>:</p> stages:</span></span> - build</span></span> - deploy</span></span> </span> default:</span></span> image: debian:stable-slim</span></span> </span> variables:</span></span> # Fetch theme submodules automatically.</span></span> GIT_SUBMODULE_STRATEGY: "recursive"</span></span> </span> # Keep CI on the same major/minor Zola as local development.</span></span> ZOLA_VERSION: "0.22.1"</span></span> </span> PANDOC_VERSION: "3.9"</span></span> TYPST_VERSION: "0.15.0"</span></span> </span> SITE_BASE_URL: "https://blog.none.at"</span></span> COMPRESS_REGEX: ".\\.(htm\|html\|xml\|txt\|text\|js\|css\|svg\|pdf)$"</span></span> </span> build-pdfs:</span></span> stage: build</span></span> script:</span></span> - \|</span></span> apt-get update</span></span> DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends \</span></span> wget ca-certificates python3 python3-yaml xz-utils \</span></span> fonts-lato fonts-hack</span></span> </span> echo "* Installing pandoc ${PANDOC_VERSION}"</span></span> PANDOC_TARBALL="pandoc-${PANDOC_VERSION}-linux-amd64.tar.gz"</span></span> wget --no-verbose "https://github.com/jgm/pandoc/releases/download/${PANDOC_VERSION}/${PANDOC_TARBALL}"</span></span> tar -xzf "${PANDOC_TARBALL}" --strip-components=2 -C /usr/local/bin pandoc-${PANDOC_VERSION}/bin/pandoc</span></span> </span> echo "* Installing typst ${TYPST_VERSION}"</span></span> TYPST_TARBALL="typst-x86_64-unknown-linux-musl.tar.xz"</span></span> wget --no-verbose "https://github.com/typst/typst/releases/download/v${TYPST_VERSION}/${TYPST_TARBALL}"</span></span> tar -xJf "${TYPST_TARBALL}" --strip-components=1 -C /usr/local/bin \</span></span> "typst-x86_64-unknown-linux-musl/typst"</span></span> </span> echo "* Building PDFs"</span></span> bash scripts/build-pdfs.sh</span></span> artifacts:</span></span> paths:</span></span> - static/pdf</span></span> expire_in: 1 week</span></span> </span> pages-build:</span></span> stage: build</span></span> needs:</span></span> - build-pdfs</span></span> script:</span></span> - \|</span></span> apt-get update</span></span> DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends wget ca-certificates python3</span></span> ZOLA_RELEASE_DIR="https://github.com/getzola/zola/releases/download/v${ZOLA_VERSION}"</span></span> ZOLA_TARBALL="zola-v${ZOLA_VERSION}-x86_64-unknown-linux-gnu.tar.gz"</span></span> echo "*****************************************************"</span></span> echo ""</span></span> echo "* Using zola version: ${ZOLA_VERSION}"</span></span> echo "* Retrieving zola tarball: ${ZOLA_TARBALL}"</span></span> echo "* ..from: ${ZOLA_RELEASE_DIR}"</span></span> echo ""</span></span> echo "****************************************************"</span></span> wget --no-verbose "${ZOLA_RELEASE_DIR}/${ZOLA_TARBALL}"</span></span> tar -xzf "${ZOLA_TARBALL}"</span></span> ./zola build --base-url "${SITE_BASE_URL}" --minify</span></span> python3 scripts/postprocess_jsonld_redirects.py public</span></span> artifacts:</span></span> paths:</span></span> - public</span></span> expire_in: 1 week</span></span> </span> pages:</span></span> stage: deploy</span></span> pages: true</span></span> dependencies:</span></span> - pages-build</span></span> script:</span></span> - \|</span></span> echo "Deploy pages built in previous stage"</span></span> apt-get update</span></span> DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes --no-install-recommends brotli zstd</span></span> find public -type f -regex "${COMPRESS_REGEX}" -exec gzip -f -k {} \;</span></span> find public -type f -regex "${COMPRESS_REGEX}" -exec brotli -f -k {} \;</span></span> find public -type f -regex "${COMPRESS_REGEX}" -exec zstd -f -k {} \;</span></span> artifacts:</span></span> paths:</span></span> - public</span></span> expire_in: 1 week</span></span> rules:</span></span> - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH</span></span></code></pre>Custom domains🔗</a></h3> Because GitLab Pages supports custom domains</a>, I configured the blog</code> subdomain for none.at</code>. That gives me a cleaner URL, as described in the GitLab Pages DNS records documentation</a>.</p> Make it public🔗</a></h2> After these steps, I run the usual Git commands to publish changes:</p> git add .</li> git commit -s -m ‘Init’</li> git push</li> </ul> I repeat this workflow for future posts with a proper commit message 😄.</p> Mail setup for deSEC 2025-07-18T00:00:00+00:00 Introduction🔗</a></h1> This post describes how to configure the mail DNS setup for deSEC</a> via commandline.</p> Prepare🔗</a></h1> General information🔗</a></h2> It’s recommended to install the certbot-dns-desec</a> because this makes the interaction with let’s encrypt (also written as “LE”) much easier.</p> In this post and for the setup are the following tools required bash</code>, curl</code>, jq</code>, envsubst</code> and certbot</code>.</p> For the beginning will we export the two environment variables below.</p> export</span> DNSSEC_TOKEN</span>=</span>'</span>Your-DNSSEC-API-Token</span>'</span></span> export</span> DOMAIN_NAME</span>=</span>'</span>Your-Domain</span>'</span></span></code></pre> The value for the DNSSEC_TOKEN</code> can be created via the Web UI or via API as documented at Creating a Token</a>.</p> Create Certificate🔗</a></h2> The creation of the file /etc/letsencrypt/secrets/$DOMAIN_NAME.ini</code> is documented on the certbot-dns-desec</a> We can now create the LE certificate and key after the preparation above.</p> warning </div> Warning</strong></span> </div> Please be aware that an wildcard certificate will be created with the call below!</p> </div> </div> certbot</span> certonly</span> \</span></span> -</span>-authenticator</span> dns-desec</span> \</span></span> -</span>-dns-desec-credentials</span> /etc/letsencrypt/secrets/</span>$</span>DOMAIN_NAME</span>.ini</span> \</span></span> -</span>d</span> '</span>$DOMAIN_NAME</span>'</span> \</span></span> -</span>d</span> '</span>.$DOMAIN_NAME</span>'</span></span></code></pre> There must be an directory /etc/letsencrypt/live/${DOMAIN_NAME}/fullchain.pem</code> after the successfully execution of the certbot call.</p> The deSEC json file🔗</a></h2> To be able to create the DNS Resource Record Sets (RRsets) is a json file required. The json file below can be used to create the RRsets via the API call, which is shown at the end. Save the json text into the file mail-setup.json</code>.</p> [</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>MX</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>5 mail.${DOMAIN_NAME}.</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TXT</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>\"</span>v=spf1 mx -all</span>\"</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>A</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>IPv4 Adress</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>AAAA</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> [</span>"</span>IPv6 Adress</span>"</span>]</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>CAA</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>0 iodef mailto:postmaster@${DOMAIN_NAME}</span>"</span>,</span></span> "</span>0 issue letsencrypt.org</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>_25._tcp.mail</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TLSA</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>3 1 1 ${TLSA_SHA}</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>_dmarc</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TXT</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>\"</span>v=DMARC1;p=quarantine;pct=100;rua=mailto:postmaster@${DOMAIN_NAME};ruf=mailto:postmaster@${DOMAIN_NAME}</span>\"</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>_mta-sts</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TXT</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>\"</span>v=STSv1; id=2023042102;</span>\"</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>_smtp._tls</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TXT</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>\"</span>v=TLSRPTv1; rua=mailto:postmaster@${DOMAIN_NAME}</span>\"</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>dkim._domainkey</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>TXT</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>\"</span>v=DKIM1; p=${DKIM_PUB_KEY}</span>\"</span>"</span></span> ]</span></span> }</span>,</span></span> {</span></span> "</span>subname</span>"</span>:</span> "</span>mta-sts</span>"</span>,</span></span> "</span>type</span>"</span>:</span> "</span>CNAME</span>"</span>,</span></span> "</span>ttl</span>"</span>:</span> 3600</span>,</span></span> "</span>records</span>"</span>:</span> [</span></span> "</span>mail.${DOMAIN_NAME}.</span>"</span></span> ]</span></span> }</span></span> ]</span></span></code></pre>Add Data to deSEC🔗</a></h1> There are now two more environment variables necessary before the call to the deSEC API is possible.</p> DKIM_PUB_KEY</code> is the dkim key from your mail system. for example with amavis /usr/sbin/amavisd showkeys ${DOMAIN_NAME}</code></li> </ul> </li> TLSA_HASH</code> is the sha256 hash for the TLSA</code> Record.</li> </ul> example call🔗</a></h2> The shell code below shows how such a shell session looks like.</p> warning </div> Example</strong></span> </div> You will need to adopt the environment variables.</p> </div> </div> export</span> DNSSEC_TOKEN</span>=</span>'</span>Your-DNSSEC-API-Token</span>'</span></span> export</span> DOMAIN_NAME</span>=</span>'</span>Your-Domain</span>'</span></span> export</span> DKIM_PUB_KEY</span>=</span>'</span>Your-DKIM-Public-Key</span>'</span></span> export</span> TLSA_HASH</span>=</span>$(</span>openssl</span> x509</span> -</span>noout</span> -</span>pubkey</span> \</span></span> -</span>in</span> /etc/letsencrypt/live/</span>$</span>{</span>DOMAIN_NAME</span>}</span>/fullchain.pem</span> \</span></span> \|</span> openssl</span> pkey</span> -</span>pubin</span> -</span>outform</span> DER</span> \</span></span> \|</span> openssl</span> sha256</span> 2</span>>&1</span> \</span></span> \|</span> cut</span> -</span>f2</span> -</span>d</span> '</span> '</span>)</span></span> </span> envsubst</span> <</span> mail-setup.json</span>\|</span>curl</span> -</span>sSLX</span> PUT</span> \</span></span> https://desec.io/api/v1/domains/</span>$</span>{</span>DOMAIN_NAME</span>}</span>/rrsets/</span> \</span></span> -</span>-header</span> "</span>Authorization: Token </span>$</span>{</span>DNSSEC_TOKEN</span>}</span>"</span> \</span></span> -</span>-header</span> "</span>Content-Type: application/json</span>"</span> -</span>-data</span> @-</span> \|</span> jq</span></span></code></pre> tls-proxy-tunnel: Transparent TLS Tunnelling Through Corporate HTTP Proxies 2024-07-02T00:00:00+00:00 Many enterprise networks sit behind a corporate HTTP CONNECT proxy. Applications that speak TLS natively — think git</code>, curl</code>, SSH-over-HTTPS, or any custom binary — often have no built-in proxy support. Configuring every single tool is tedious, fragile, and sometimes impossible when you don’t control the binary.</p> tls-proxy-tunnel</code> (tpt</code>) solves this at layer 4: it sits between your application and the outside world, intercepts the TLS connection, extracts the Server Name Indication (SNI) from the ClientHello, and tunnels the raw bytes through your corporate HTTP CONNECT proxy — without ever terminating TLS.</p> Building a native file upload handler for Caddy v2 2022-03-01T00:00:00+00:00 When Caddy v2</a> was released in 2020, it introduced a new concept of handlers that use Go’s native module features.</p> In 2022 one of our customers needed to upload files, and at that time none of the available web servers could handle this natively without an extra program. So I decided to create a new handler for Caddy v2 — caddyv2-upload</a> was born.</p> I was not aware of any other tool which can handle file uploads natively in just one binary which can be used in Kubernetes or OpenShift, or I have not searched hard enough 😁. I also wanted to learn go, so I decided to create a new handler for the go based webserver caddyserver</a>.</p> In May 2022 dufs</a>, a Rust-based server, was created — so that might be another option worth considering. 🤷</p> Any how I wanted to learn go in the pre AI Area and at that time I looked into the docs of go modules, into the source of caddy servers own handlers and created step by step a new handler.</p> Luckily there was already podman and buildah, so I was able to create container images to run and test locally.</p> Building something real turned out to be the most effective way to learn Go. Diving into Caddy’s own handler source code, understanding the module system, and iterating with containers made the concepts stick far better than any tutorial would have.</p> A minimal Caddyfile configuration looks like this:</p> {</span></span> order upload before file_server</span></span> }</span></span> </span> :2015 {</span></span> root .</span></span> </span> file_server browse</span></span> </span> @mypost method POST</span></span> upload @mypost {</span></span> dest_dir /var/uploads</span></span> max_filesize 4MB</span></span> }</span></span> }</span></span></code></pre> Build with xcaddy</a>:</p> xcaddy</span> build</span> -</span>-with</span> github.com/git001/caddyv2-upload</span></span></code></pre> Today caddyv2-upload</a> (v0.20.0) supports file uploads via HTTP multipart form, configurable destination directories, UUID-based subdirectory creation per upload, fixed filenames, pass-through mode to continue to the next Caddy handler after upload, configurable file size limits, an HTTP notification callback after successful uploads, and a /health</code> endpoint. It integrates with Caddy’s variable system and works in Kubernetes and OpenShift environments. Authentication is handled by Caddy’s own security layer rather than the module itself.</p> How does SNI Routing work in HAProxy 2019-05-17T00:00:00+00:00 As I travel a lot I faced the problem that in some Wi-Fi networks certain ports are blocked for outgoing communication 😱. The solution is to use software that can handle TCP and HTTP via port 443</strong> so that I can use my Nextcloud and my XMPP client on the same port.</p>

Which Tool Mirrors a Cosign-Signed Image into a Private Registry?

haproxy-spoe-rs: Deployment

Running pdns_recursor as a root-independent validating resolver

Running a validating DNS recursor from the root zone with Hickory DNS

vigil-log-relay: Kubernetes Log Collection Without a DaemonSet

vigil-rs: A Rust Service Supervisor for Containers

mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster

AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser

AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers

loadgen-rs (Short): Results, Quick Start, and Tool Choice

loadgen-rs: An HTTP Benchmark Client in Rust

cert-manager-webhook-libdns: One Webhook, Many DNS Providers

Simple autotls setup with ferron

Get Token.🔗</a></h1>
To be able to add the acme challenge to the deSEC is a Token required. How to get and mange tokens for deSEC is documented at Manage Tokens</a></p>

How to automate TLSA update

DeSEC, domain + mail setup 🔗</a></h1>
The “Mail setup for deSEC” is described at the link Mail setup for deSEC</a>.</p>

certbot 🔗</a></h1>
Certbot offers the option to run some hooks at certificate handling Pre and Post Validation Hooks</a>.</p>

`Finally🔗</a></h1> Wenn everything works as designed should you never need to update the TLSA record when the LE certificate was changed.</p>`

How to Run Tor Arti as an HTTP Proxy or Behind a g3 HTTP Proxy

`Update🔗</a></h2> Since Arti 1.7.0 (October 30, 2025)</a>, HTTP CONNECT</code> is supported. That means Arti can be used as an HTTP proxy endpoint, not only as a SOCKS5 endpoint.</p>`

How to Host a Zola-Generated Site on GitLab Pages

Custom domains 🔗</a></h3>
Because GitLab Pages supports custom domains</a>, I configured the `blog</code> subdomain for none.at</code>. That gives me a cleaner URL, as described in theGitLab Pages DNS records documentation</a>.</p>`

`Make it public🔗</a></h2> After these steps, I run the usual Git commands to publish changes:</p> git add .</li> git commit -s -m ‘Init’</li> git push</li> </ul> I repeat this workflow for future posts with a proper commit message 😄.</p>`

Mail setup for deSEC

Introduction 🔗</a></h1>
This post describes how to configure the mail DNS setup for deSEC</a> via commandline.</p>

tls-proxy-tunnel: Transparent TLS Tunnelling Through Corporate HTTP Proxies

Building a native file upload handler for Caddy v2

How does SNI Routing work in HAProxy

Alek's Blog - open-source

k8s-scale-app-rs: Scale or Restart a Kubernetes Deployment from a CronJob

Alek's Blog - open-source

k8s-scale-app-rs: Scale or Restart a Kubernetes Deployment from a CronJob

Which Tool Mirrors a Cosign-Signed Image into a Private Registry?

haproxy-spoe-rs: Deployment

Running pdns_recursor as a root-independent validating resolver

Running a validating DNS recursor from the root zone with Hickory DNS

vigil-log-relay: Kubernetes Log Collection Without a DaemonSet

vigil-rs: A Rust Service Supervisor for Containers

mergelog-rs: Rewriting a Year-2000 C Tool in Rust — and Making It 2.26× Faster

AWFFull 4.0.0 — Modernising a 2008 Web Log Analyser

AWFFull in the Cloud: Shipping Logs to Object Storage and Running in Containers

loadgen-rs (Short): Results, Quick Start, and Tool Choice

loadgen-rs: An HTTP Benchmark Client in Rust

Important fairness fix🔗</a></h3> One easy mistake is RPS interpretation:</p> In loadgen-rs, --rps</code> is per client</strong></li>