https://blog.none.at Production notes on Kubernetes, OpenShift, and OVHcloud: observability, log archiving, service mesh, LLM inference, and digital sovereignty. Zola en Tue, 03 Mar 2026 00:00:00 +0000 Tue, 03 Mar 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-03-01-loadgen-rs-short/ https://blog.none.at/blog/2026/2026-03-01-loadgen-rs-short/ <p>This is the short version.<br /> If you want profiling details, FFI internals, and full cloud deployment walkthroughs, read the full article: <a rel="noopener external" target="_blank" href="https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/">loadgen-rs: An HTTP Benchmark Client in Rust</a>.</p> <h2 id="tl-dr">TL;DR<a class="zola-anchor" href="#tl-dr" aria-label="Anchor link for: tl-dr">🔗</a></h2> <p><code>loadgen-rs</code> is a Rust benchmark client for HTTP/1.1, HTTP/2, and HTTP/3 with:</p> <ul> <li>one CLI for all protocols</li> <li>JSONL/CSV output for automation</li> <li>distributed controller/worker mode</li> <li>scripted Deno scenarios via FFI (k6-style checks/extractors)</li> </ul> <p>In short: very strong for reproducible automation and multi-node testing.<br /> If your only goal is absolute H3 peak throughput, <code>h2load</code> still has an edge.</p> <h2 id="quick-start">Quick Start<a class="zola-anchor" href="#quick-start" aria-label="Anchor link for: quick-start">🔗</a></h2> <h3 id="single-node-cli">Single-node cli<a class="zola-anchor" href="#single-node-cli" aria-label="Anchor link for: single-node-cli">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> H2 example, 10s run</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span></code></pre><h3 id="single-node-container">Single-node container<a class="zola-anchor" href="#single-node-container" aria-label="Anchor link for: single-node-container">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTP/3 (QUIC) 10 seconds</span></span> <span class="giallo-l"><span class="z-entity z-name">podman</span><span class="z-string"> run</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">it</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rm</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-network</span><span class="z-string"> host</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-name</span><span class="z-string"> gh-loadgen</span><span class="z-string"> ghcr.io/git001/loadgen-rs:v0.3.0</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h3</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span></code></pre><h3 id="distributed">Distributed<a class="zola-anchor" href="#distributed" aria-label="Anchor link for: distributed">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1) start workers</span></span> <span class="giallo-l"><span class="z-entity z-name">bash</span><span class="z-string"> examples/worker-start.sh</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2) run controller</span></span> <span class="giallo-l"><span class="z-entity z-name">deno</span><span class="z-string"> run</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-ffi</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-net</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-read</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-env</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> examples/distributed.ts</span><span class="z-string"> http://worker1:9091</span><span class="z-string"> http://worker2:9091</span></span></code></pre> <p>Why distributed matters: you can scale beyond one machine and still get statistically correct merged percentiles (histogram merge, not naive p99 averaging).</p> <h2 id="benchmark-snapshot-vs-h2load">Benchmark Snapshot vs h2load<a class="zola-anchor" href="#benchmark-snapshot-vs-h2load" aria-label="Anchor link for: benchmark-snapshot-vs-h2load">🔗</a></h2> <p>High-concurrency test (<code>-c 512 -t 8 -m 8</code>, 30s):</p> <table><thead><tr><th>Protocol</th><th style="text-align: right">h2load RPS</th><th style="text-align: right">loadgen-rs RPS</th><th style="text-align: right">Ratio</th></tr></thead><tbody> <tr><td>H1</td><td style="text-align: right">18,851</td><td style="text-align: right">19,148</td><td style="text-align: right">101.6%</td></tr> <tr><td>H2</td><td style="text-align: right">17,837</td><td style="text-align: right">18,025</td><td style="text-align: right">101.1%</td></tr> <tr><td>H3</td><td style="text-align: right">9,006</td><td style="text-align: right">6,702</td><td style="text-align: right">74.4%</td></tr> </tbody></table> <p>Takeaway:</p> <ul> <li>H1/H2 throughput: effectively on par (or slightly better for loadgen-rs in this run).</li> <li>H3 throughput: h2load remains faster.</li> <li>Memory footprint: h2load is smaller; loadgen-rs typically used more memory in tests.</li> </ul> <h2 id="which-tool-should-you-use">Which Tool Should You Use?<a class="zola-anchor" href="#which-tool-should-you-use" aria-label="Anchor link for: which-tool-should-you-use">🔗</a></h2> <table><thead><tr><th>Need</th><th>Better fit</th></tr></thead><tbody> <tr><td>Highest H3 throughput only</td><td>h2load</td></tr> <tr><td>One tool for H1/H2/H3 + machine-readable reports</td><td>loadgen-rs</td></tr> <tr><td>Distributed multi-machine load generation</td><td>loadgen-rs</td></tr> <tr><td>Scripted scenarios with checks/extractors and correlation</td><td>loadgen-rs</td></tr> <tr><td>Lowest memory footprint</td><td>h2load</td></tr> </tbody></table> <h2 id="next-step">Next Step<a class="zola-anchor" href="#next-step" aria-label="Anchor link for: next-step">🔗</a></h2> <p>For the complete deep dive (profiling, architecture, FFI API design, Terraform/Ansible workflow), continue with the full article: <a rel="noopener external" target="_blank" href="https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/">loadgen-rs: An HTTP Benchmark Client in Rust</a>.</p> Tue, 03 Mar 2026 00:00:00 +0000 aleks https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/ https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/ <h2 id="tl-dr-executive-summary">TL;DR / Executive Summary<a class="zola-anchor" href="#tl-dr-executive-summary" aria-label="Anchor link for: tl-dr-executive-summary">🔗</a></h2> <p>Prefer a shorter read first?<br /> <a rel="noopener external" target="_blank" href="https://blog.none.at/blog/2026/2026-03-01-loadgen-rs-short/">loadgen-rs (Short): Results, Quick Start, and Tool Choice</a></p> <p>loadgen-rs is a Rust HTTP benchmark client that covers HTTP/1.1, HTTP/2, and HTTP/3 in one tool, outputs machine-readable JSONL/CSV, and scales from single-node runs to distributed multi-worker tests.</p> <ul> <li>H1/H2 throughput is competitive with h2load, while H3 still favors h2load in peak throughput.</li> <li>loadgen-rs is strong when you want automation-friendly output and one consistent workflow across protocols.</li> <li>Distributed mode is a first-class feature, not an afterthought: controller + workers + correct histogram merge.</li> <li>A Deno/FFI layer enables scripted, k6-style scenarios with checks, extractors, and correlation.</li> </ul> <h2 id="what-is-loadgen-rs">What Is loadgen-rs?<a class="zola-anchor" href="#what-is-loadgen-rs" aria-label="Anchor link for: what-is-loadgen-rs">🔗</a></h2> <h3 id="why-i-built-it">Why I Built It<a class="zola-anchor" href="#why-i-built-it" aria-label="Anchor link for: why-i-built-it">🔗</a></h3> <p>If you benchmark HTTP servers, you probably know <a rel="noopener external" target="_blank" href="https://nghttp2.org/documentation/h2load-howto.html">h2load</a> from the nghttp2 project. It’s a very solid and mature tool, but there are a few pain points:</p> <ul> <li><strong>HTTP/3 support is not enabled by default</strong> — h2load can do H3/QUIC, but only via a separate build variant with ngtcp2.</li> <li><strong>Requires a C compiler and many native libraries</strong> — this makes source builds and CI setup more cumbersome, especially across different environments.</li> </ul> <p>I wanted a tool that tests all three HTTP protocols and outputs the results directly as JSON and <em>easy</em> to build. So I built <strong><a rel="noopener external" target="_blank" href="https://github.com/git001/loadgen-rs">loadgen-rs</a></strong>.</p> <h3 id="core-capabilities-h1-h2-h3-jsonl-distributed-scripting">Core Capabilities (H1/H2/H3, JSONL, Distributed, Scripting)<a class="zola-anchor" href="#core-capabilities-h1-h2-h3-jsonl-distributed-scripting" aria-label="Anchor link for: core-capabilities-h1-h2-h3-jsonl-distributed-scripting">🔗</a></h3> <table><thead><tr><th>Component</th><th>Crate</th><th>Why</th></tr></thead><tbody> <tr><td>Runtime</td><td>tokio (multi-thread)</td><td>De-facto standard for async Rust</td></tr> <tr><td>HTTP/1.1</td><td>Raw TCP + httparse</td><td>Maximum control, zero framework overhead</td></tr> <tr><td>HTTP/2</td><td>h2 (direct, no hyper)</td><td>Avoids hyper’s abstraction layer</td></tr> <tr><td>TLS</td><td>rustls (ring backend)</td><td>Pure Rust, no OpenSSL dependency hell</td></tr> <tr><td>HTTP/3</td><td>quinn + h3 + h3-quinn</td><td>QUIC stack in Rust</td></tr> <tr><td>CLI</td><td>clap (derive)</td><td>Ergonomic, type-safe</td></tr> <tr><td>Metrics</td><td>hdrhistogram</td><td>Industry standard for latency measurements</td></tr> <tr><td>Allocator</td><td>mimalloc</td><td>Better multi-threaded allocation performance</td></tr> </tbody></table> <p>An early design choice was to bypass <a rel="noopener external" target="_blank" href="https://hyper.rs/">hyper</a> entirely. For H1, we use raw TCP sockets with <code>httparse</code> for response parsing. For H2, we use the <code>h2</code> crate directly. This eliminates one abstraction layer and gives full control over connection management, flow control tuning, and request templating. <a rel="noopener external" target="_blank" href="https://hyper.rs/">Hyper</a> is a great library, but its abstraction model didn’t fit the needs of this benchmarking tool.</p> <p>In addition to single-node CLI benchmarking, loadgen-rs includes distributed controller/worker execution and scripted scenario support through a native FFI layer for Deno.</p> <h3 id="runtime-and-execution-model">Runtime and Execution Model<a class="zola-anchor" href="#runtime-and-execution-model" aria-label="Anchor link for: runtime-and-execution-model">🔗</a></h3> <figure class="mt-4"> <img class="block image-shortcode mx-auto" src="https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/runtime-and-execution-model.png" alt="Runtime and execution model: main, runner, drivers, metrics, output" style="display:block; width:100%; max-width:700px; height:auto; margin:0 auto;" loading="lazy" decoding="async" > </figure> <h4 id="connection-strategy">Connection Strategy<a class="zola-anchor" href="#connection-strategy" aria-label="Anchor link for: connection-strategy">🔗</a></h4> <p>The <code>-c</code> and <code>-m</code> flags follow h2load semantics:</p> <ul> <li><strong><code>-c N</code></strong> spawns N parallel connections distributed across worker threads</li> <li><strong><code>-m N</code></strong> limits concurrent in-flight requests per connection (H2/H3: concurrent streams; H1: in-flight cap)</li> <li>Total parallelism: <code>c * m</code></li> </ul> <p>Each worker thread runs its own <code>current_thread</code> Tokio runtime — <strong>no cross-thread synchronization in the hot path</strong>. Workers own local HdrHistograms that are merged only at the end.</p> <h4 id="count-vs-duration-mode">Count vs. Duration Mode<a class="zola-anchor" href="#count-vs-duration-mode" aria-label="Anchor link for: count-vs-duration-mode">🔗</a></h4> <ul> <li><strong>Count mode</strong> (<code>-n 1000</code>): A shared <code>AtomicU64</code> budget that workers decrement via CAS</li> <li><strong>Duration mode</strong> (<code>-D 10s</code>): A <code>CancellationToken</code> that fires after the specified duration</li> <li><code>-D</code> takes priority over <code>-n</code> (matching h2load behavior)</li> </ul> <h4 id="rps-semantics-h2load-compatible">RPS Semantics (h2load-compatible)<a class="zola-anchor" href="#rps-semantics-h2load-compatible" aria-label="Anchor link for: rps-semantics-h2load-compatible">🔗</a></h4> <p><code>--rps</code> is implemented as a <strong>per-client</strong> request start rate, matching <code>h2load</code>.</p> <ul> <li>Per-client target: <code>--rps</code></li> <li>Aggregate target: <code>--rps * -c</code></li> </ul> <p>This matters for comparisons:</p> <ul> <li><code>-c 10 --rps 20</code> targets roughly <code>200</code> started req/s overall</li> <li><code>-c 10 --rps 200</code> targets roughly <code>2000</code> started req/s overall</li> </ul> <p>Also note: in short <strong>count mode</strong> runs (<code>-n</code>), achieved <code>%</code> can appear above or below 100% because startup and shutdown overhead dominate a short elapsed window. For stable pacing comparisons, use <strong>duration mode</strong>.</p> <h3 id="output-and-tls-notes">Output and TLS Notes<a class="zola-anchor" href="#output-and-tls-notes" aria-label="Anchor link for: output-and-tls-notes">🔗</a></h3> <p>The JSONL output contains everything a Python script needs:</p> <pre class="giallo z-code"><code data-lang="json"><span class="giallo-l"><span>{</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">proto</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">h2</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">url</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">https://target:8443/</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">clients</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">threads</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">rps</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 12543.7</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">latency_p50_us</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 320</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">latency_p90_us</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 890</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">latency_p99_us</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 2100</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">status_counts</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span> {</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">200</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 125437</span><span>}</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">status_2xx</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 125437</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">err_total</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">elapsed_s</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 10.001</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>In Python:</p> <pre class="giallo z-code"><code data-lang="python"><span class="giallo-l"><span class="z-keyword">import</span><span> json</span></span> <span class="giallo-l"><span>results</span><span class="z-keyword z-keyword z-operator"> =</span><span> [</span><span>json</span><span class="z-punctuation z-separator">.</span><span class="z-meta z-function-call z-generic">loads</span><span>(</span><span>line</span><span>)</span><span class="z-keyword"> for</span><span> line</span><span class="z-keyword"> in</span><span class="z-support z-support z-function"> open</span><span>(</span><span class="z-string">&quot;</span><span class="z-string">results.jsonl</span><span class="z-string">&quot;</span><span>)</span><span>]</span></span> <span class="giallo-l"><span>df</span><span class="z-keyword z-keyword z-operator"> =</span><span> pd</span><span class="z-punctuation z-separator">.</span><span class="z-meta z-function-call z-generic">DataFrame</span><span>(</span><span>results</span><span>)</span></span> <span class="giallo-l"><span>df</span><span class="z-punctuation z-separator">.</span><span class="z-meta z-function-call z-generic">plot</span><span>(</span><span class="z-variable z-variable z-parameter z-function-call">x</span><span class="z-keyword z-keyword z-operator">=</span><span class="z-string">&quot;</span><span class="z-string">clients</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-variable z-variable z-parameter z-function-call"> y</span><span class="z-keyword z-keyword z-operator">=</span><span class="z-string">&quot;</span><span class="z-string">rps</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span><span class="z-variable z-variable z-parameter z-function-call"> kind</span><span class="z-keyword z-keyword z-operator">=</span><span class="z-string">&quot;</span><span class="z-string">bar</span><span class="z-string">&quot;</span><span>)</span></span></code></pre><h4 id="tls-configuration">TLS Configuration<a class="zola-anchor" href="#tls-configuration" aria-label="Anchor link for: tls-configuration">🔗</a></h4> <p><code>--insecure</code> implements a custom <code>ServerCertVerifier</code> that accepts everything. <code>--tls-ciphers</code> allows selecting specific cipher suites — with validation that H3 only permits TLS 1.3 ciphers. Custom CA certificates can be loaded via <code>--tls-ca</code>.</p> <h2 id="quick-start">Quick Start<a class="zola-anchor" href="#quick-start" aria-label="Anchor link for: quick-start">🔗</a></h2> <h3 id="single-node-benchmark-in-30-seconds-cli">Single-Node Benchmark in 30 Seconds (cli)<a class="zola-anchor" href="#single-node-benchmark-in-30-seconds-cli" aria-label="Anchor link for: single-node-benchmark-in-30-seconds-cli">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTP/1.1, 1000 requests</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 1000</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-string"> &#39;</span><span class="z-string">http://bench.local:8081/?s=256k</span><span class="z-string">&#39;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTPS</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 1000</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTP/2, 10 seconds</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTP/3 (QUIC)</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 2</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h3</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span></code></pre><h3 id="single-node-benchmark-in-30-seconds-container">Single-Node Benchmark in 30 Seconds (container)<a class="zola-anchor" href="#single-node-benchmark-in-30-seconds-container" aria-label="Anchor link for: single-node-benchmark-in-30-seconds-container">🔗</a></h3> <p>If you don’t have <code>podman</code>, you can use <code>docker</code> instead.</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> HTTP/3 (QUIC) 10 seconds</span></span> <span class="giallo-l"><span class="z-entity z-name">podman</span><span class="z-string"> run</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">it</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rm</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-network</span><span class="z-string"> host</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-name</span><span class="z-string"> gh-loadgen</span><span class="z-string"> ghcr.io/git001/loadgen-rs:v0.3.0</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h3</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> &#39;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&#39;</span></span></code></pre><h3 id="distributed-benchmark-in-2-minutes">Distributed Benchmark in 2 Minutes<a class="zola-anchor" href="#distributed-benchmark-in-2-minutes" aria-label="Anchor link for: distributed-benchmark-in-2-minutes">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Start workers (one per machine, or locally for testing)</span></span> <span class="giallo-l"><span class="z-entity z-name">bash</span><span class="z-string"> examples/worker-start.sh</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Run controller</span></span> <span class="giallo-l"><span class="z-entity z-name">deno</span><span class="z-string"> run</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-ffi</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-net</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-read</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-allow-env</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> examples/distributed.ts</span><span class="z-string"> http://worker1:9091</span><span class="z-string"> http://worker2:9091</span></span></code></pre> <p>Or programmatically:</p> <pre class="giallo z-code"><code data-lang="typescript"><span class="giallo-l"><span class="z-keyword">import</span><span class="z-meta z-block"> {</span><span class="z-variable z-other z-variable"> Controller</span><span class="z-meta z-block"> }</span><span class="z-keyword"> from</span><span class="z-string"> &quot;</span><span class="z-string">./ts/mod.ts</span><span class="z-string">&quot;</span><span class="z-punctuation z-terminator">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-storage">const</span><span class="z-variable z-other z-constant z-variable"> controller</span><span class="z-keyword z-keyword z-operator"> =</span><span class="z-keyword z-keyword z-operator"> new</span><span class="z-entity z-name z-function"> Controller</span><span>(</span><span>{</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> workers</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-string">&quot;</span><span class="z-string">http://worker1:9091</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span><span class="z-string"> &quot;</span><span class="z-string">http://worker2:9091</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member">]</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> config</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> url</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">https://target:8443/</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> protocol</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">h2</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> clients</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 200</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span><span class="z-punctuation z-definition z-comment z-comment"> //</span><span class="z-comment"> split: 100 per worker</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> threads</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> duration_s</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 30</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> insecure</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span>}</span><span>)</span><span class="z-punctuation z-terminator">;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-storage z-type z-storage">const</span><span class="z-variable z-other z-constant z-variable"> result</span><span class="z-keyword z-keyword z-operator"> =</span><span class="z-keyword"> await</span><span class="z-variable z-other z-variable"> controller</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">run</span><span>(</span><span>)</span><span class="z-punctuation z-terminator">;</span></span> <span class="giallo-l"><span class="z-variable z-other z-variable">console</span><span class="z-punctuation z-accessor">.</span><span class="z-entity z-name z-function">log</span><span>(</span><span class="z-variable z-other z-variable">result</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-variable">merged_report</span><span class="z-punctuation z-accessor">.</span><span class="z-variable z-other z-variable">rps</span><span>)</span><span class="z-punctuation z-terminator">;</span></span></code></pre><h3 id="minimal-comparison-against-h2load">Minimal Comparison Against h2load<a class="zola-anchor" href="#minimal-comparison-against-h2load" aria-label="Anchor link for: minimal-comparison-against-h2load">🔗</a></h3> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1) H1 count mode (same request budget)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 200</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 200</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients * 20 rps/client)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3) H2 duration mode with pacing (same effective target)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list</span><span class="z-string"> h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4) H3 duration mode with pacing (same effective target)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h3</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list</span><span class="z-string"> h3</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span></code></pre><h2 id="key-results-vs-h2load">Key Results vs h2load<a class="zola-anchor" href="#key-results-vs-h2load" aria-label="Anchor link for: key-results-vs-h2load">🔗</a></h2> <p>After implementing and tuning the tool, I ran systematic A/B comparisons against <code>h2load</code> on the same hardware and target server.</p> <h3 id="high-concurrency-benchmark">High-Concurrency Benchmark<a class="zola-anchor" href="#high-concurrency-benchmark" aria-label="Anchor link for: high-concurrency-benchmark">🔗</a></h3> <p>Setup:</p> <ul> <li>Duration: 30 seconds per run, 30-second cooldown between runs</li> <li>Configuration: <code>-c 512 -t 8 -m 8</code></li> <li>Target: <code>https://127.0.0.1:8082/?s=256k</code> (256 KB response body)</li> <li>h2load runs via <code>podman run --rm --network host localhost/h2load</code></li> </ul> <table><thead><tr><th>Protocol</th><th>h2load RPS</th><th>loadgen-rs RPS</th><th>Ratio</th><th>h2load Latency (mean)</th><th>loadgen-rs Latency (mean)</th></tr></thead><tbody> <tr><td><strong>H1</strong></td><td>18,851</td><td><strong>19,148</strong></td><td><strong>101.6%</strong></td><td>27.07 ms</td><td>25.69 ms</td></tr> <tr><td><strong>H2</strong></td><td>17,837</td><td><strong>18,025</strong></td><td><strong>101.1%</strong></td><td>125.59 ms</td><td>223.91 ms</td></tr> <tr><td><strong>H3</strong></td><td><strong>9,006</strong></td><td>6,702</td><td>74.4%</td><td>450.15 ms</td><td>604.17 ms</td></tr> </tbody></table> <p>Key observations:</p> <ul> <li><strong>H1</strong>: loadgen-rs is <strong>faster</strong> than h2load on both throughput and latency. The raw TCP + httparse approach with pre-built request templates pays off.</li> <li><strong>H2</strong>: Throughput is essentially identical. However, mean latency is ~1.8x higher in loadgen-rs due to an architectural difference (see profiling section below).</li> <li><strong>H3</strong>: h2load (using ngtcp2/nghttp3 in C) retains a ~25% throughput advantage over quinn/h3 in Rust.</li> </ul> <p>To be fair, h2load was sometimes better than loadgen-rs, and vice versa.</p> <h3 id="resource-usage-cpu-and-memory">Resource Usage (CPU and Memory)<a class="zola-anchor" href="#resource-usage-cpu-and-memory" aria-label="Anchor link for: resource-usage-cpu-and-memory">🔗</a></h3> <p>For resource fairness, both tools were run as containers and sampled with <code>podman stats</code>:</p> <p>Setup:</p> <ul> <li>3 repeats per point, <code>duration=5s</code>, <code>c=4</code>, <code>t=2</code>, <code>m in {1,8}</code></li> </ul> <table><thead><tr><th>Metric</th><th>H1/H2</th><th>H3</th></tr></thead><tbody> <tr><td>CPU time</td><td>mostly <code>+2%</code> to <code>+8%</code></td><td><code>-1%</code> to <code>+10%</code></td></tr> <tr><td>Peak memory</td><td>about <code>+410%</code> to <code>+914%</code></td><td>about <code>+1393%</code></td></tr> </tbody></table> <p>In absolute terms, peak memory was typically:</p> <ul> <li><code>h2load</code>: around <code>2.3 MB</code> to <code>3.0 MB</code></li> <li><code>loadgen-rs</code>: around <code>15 MB</code> to <code>39 MB</code></li> </ul> <p>The higher memory usage comes from Tokio’s runtime overhead, per-worker HdrHistograms, and Rust’s buffered I/O layers. In practice, even 39 MB is negligible for a benchmark client.</p> <h2 id="which-tool-should-you-choose">Which Tool Should You Choose?<a class="zola-anchor" href="#which-tool-should-you-choose" aria-label="Anchor link for: which-tool-should-you-choose">🔗</a></h2> <h3 id="decision-matrix-by-use-case">Decision Matrix by Use Case<a class="zola-anchor" href="#decision-matrix-by-use-case" aria-label="Anchor link for: decision-matrix-by-use-case">🔗</a></h3> <table><thead><tr><th>Priority</th><th>Recommendation</th></tr></thead><tbody> <tr><td>H1/H2 throughput + automation</td><td><strong>loadgen-rs</strong> — matches or beats h2load, with native JSONL output</td></tr> <tr><td>H3 peak throughput</td><td><strong>h2load</strong> — ngtcp2 is ~25% faster than quinn</td></tr> <tr><td>Lowest memory footprint</td><td><strong>h2load</strong> — 3 MB vs 15-39 MB</td></tr> <tr><td>Cross-protocol consistency</td><td><strong>loadgen-rs</strong> — same tool, same output format for H1/H2/H3</td></tr> <tr><td>Multi-machine load generation</td><td><strong>loadgen-rs</strong> — built-in distributed mode with correct histogram merge</td></tr> <tr><td>Cloud-provisioned benchmarks</td><td><strong>loadgen-rs</strong> — Terraform + Ansible pipeline for Hetzner Cloud</td></tr> <tr><td>Distributed scripted scenarios</td><td><strong>loadgen-rs</strong> — workers run multi-step checks/extractors via <code>POST /run-script</code></td></tr> </tbody></table> <h3 id="practical-recommendation">Practical Recommendation<a class="zola-anchor" href="#practical-recommendation" aria-label="Anchor link for: practical-recommendation">🔗</a></h3> <p>For most benchmarking workflows, the choice comes down to this: do you need the absolute highest H3 throughput, or do you want a single binary with all protocols available out of the box?</p> <h2 id="distributed-mode-overview">Distributed Mode Overview<a class="zola-anchor" href="#distributed-mode-overview" aria-label="Anchor link for: distributed-mode-overview">🔗</a></h2> <p>A single loadgen-rs process can saturate most test targets, but sometimes you need more — testing a load balancer with geographically distributed traffic, generating more requests than one machine’s NIC can handle, or simply simulating a more realistic multi-source access pattern.</p> <h3 id="architecture">Architecture<a class="zola-anchor" href="#architecture" aria-label="Anchor link for: architecture">🔗</a></h3> <p>The distributed mode uses a TypeScript controller (running in Deno) that orchestrates multiple worker-agents over plain HTTP/JSON — no gRPC or custom protocol needed:</p> <figure class="mt-4"> <img class="block image-shortcode mx-auto" src="https://blog.none.at/blog/2026/2026-03-01-loadgen-rs/architecture.png" alt="Distributed architecture: controller and workers with report merge" style="display:block; width:100%; max-width:700px; height:auto; margin:0 auto;" loading="lazy" decoding="async" > </figure> <p>Each worker is a lightweight Deno HTTP server (~120 LOC) that wraps the same <code>LoadgenFFI</code> batch API used for single-node benchmarks. The controller splits the total client count (<code>-c</code>) evenly across workers, sets a coordinated start timestamp, and fires all jobs concurrently.</p> <h3 id="the-histogram-merge-problem">The histogram merge problem<a class="zola-anchor" href="#the-histogram-merge-problem" aria-label="Anchor link for: the-histogram-merge-problem">🔗</a></h3> <p>Merging latency results from distributed workers is deceptively tricky. The naive approach — averaging p99 values from each worker — is <strong>statistically wrong</strong>. If worker A has p99 = 5ms (from 50K requests) and worker B has p99 = 15ms (from 50K requests), the combined p99 is <em>not</em> 10ms. It depends on the full distribution shape.</p> <p>The correct approach is to merge the underlying histograms. Each worker serializes its HdrHistogram using V2-Deflate format (a compact binary encoding, typically 1–5 KB per histogram), base64-encodes it, and includes it in the <code>RunReport</code> JSON:</p> <pre class="giallo z-code"><code data-lang="json"><span class="giallo-l"><span>{</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">rps</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 136949.0</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">latency_p99_us</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 210</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">latency_hist_b64</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">HISTFAAAB...</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">ttfb_hist_b64</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">HISTFAAAB...</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-support z-type z-property-name z-json z-support z-type z-property-name"> &quot;</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">connect_hist_b64</span><span class="z-support z-type z-property-name z-json z-support z-type z-property-name">&quot;</span><span class="z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">HISTFAAAB...</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>The controller passes all worker reports to <code>loadgen_merge_reports()</code> — a new FFI export that deserializes each histogram, merges them with <code>Histogram::add()</code>, and recomputes all percentile fields from the combined distribution. This is the same merge algorithm used internally for per-thread metrics within a single process.</p> <h3 id="coordinated-start">Coordinated start<a class="zola-anchor" href="#coordinated-start" aria-label="Anchor link for: coordinated-start">🔗</a></h3> <p>For benchmark results to be meaningful, all workers should start at the same time. The controller calculates <code>start_at = now + 500ms</code>, includes it in each worker’s <code>/run</code> request, and each worker sleeps until that timestamp before starting the benchmark. Clock skew under 10ms is acceptable for benchmark purposes — and on a local network or the same machine, skew is typically sub-millisecond.</p> <h3 id="minimal-new-code-maximum-reuse">Minimal new code, maximum reuse<a class="zola-anchor" href="#minimal-new-code-maximum-reuse" aria-label="Anchor link for: minimal-new-code-maximum-reuse">🔗</a></h3> <p>The implementation reuses the existing FFI infrastructure almost entirely:</p> <table><thead><tr><th>Component</th><th>New code</th><th>Reused</th></tr></thead><tbody> <tr><td>Rust: <code>merge_distributed_reports()</code></td><td>~120 LOC</td><td>Existing <code>merge_metrics()</code> pattern</td></tr> <tr><td>Rust: FFI export</td><td>~25 LOC</td><td>Existing <code>catch_unwind</code> + error slot pattern</td></tr> <tr><td>TypeScript: worker-agent</td><td>~120 LOC</td><td><code>LoadgenFFI</code> class</td></tr> <tr><td>TypeScript: Controller</td><td>~150 LOC</td><td><code>mergeReports()</code> FFI wrapper</td></tr> </tbody></table> <p>Only two new Rust dependencies were needed: <code>base64</code> (for histogram serialization) was already an indirect dependency via reqwest. The <code>export_histograms</code> field in <code>BenchConfig</code> defaults to <code>false</code>, so the existing CLI and single-node FFI path are completely unaffected.</p> <h3 id="real-world-validation">Real-world validation<a class="zola-anchor" href="#real-world-validation" aria-label="Anchor link for: real-world-validation">🔗</a></h3> <p>Testing 2 local workers against the same target (H2, 8 total clients, 5s):</p> <table><thead><tr><th>Setup</th><th>RPS</th><th>p99 Latency</th></tr></thead><tbody> <tr><td>Single-node (<code>-c 8 -m 4</code>)</td><td>284,226</td><td>185 us</td></tr> <tr><td>Distributed (2 × 4 clients)</td><td>269,351</td><td>207 us</td></tr> <tr><td>Ratio</td><td><strong>94.8%</strong></td><td></td></tr> </tbody></table> <p>The ~5% overhead comes from the HTTP/JSON coordination round-trip and the coordinated start delay. In a real multi-machine deployment (where each worker has its own NIC and CPU), total throughput scales linearly — you’re no longer bound by a single machine’s resources.</p> <h2 id="scripted-scenarios-overview-deno-k6-style">Scripted Scenarios Overview (Deno / k6-style)<a class="zola-anchor" href="#scripted-scenarios-overview-deno-k6-style" aria-label="Anchor link for: scripted-scenarios-overview-deno-k6-style">🔗</a></h2> <p>After the initial release, I added a TypeScript/Deno layer on top of the native FFI to make scripted testing and k6-style output easier to compare.</p> <h3 id="what-was-added">What was added<a class="zola-anchor" href="#what-was-added" aria-label="Anchor link for: what-was-added">🔗</a></h3> <ul> <li>A typed Deno SDK in <code>ts/</code>: <ul> <li><code>LoadgenFFI</code> for run-level benchmarking</li> <li><code>LoadgenStepFFI</code> for step/session execution</li> </ul> </li> <li>A reusable <code>printK6LikeSummary()</code> formatter used by the Deno examples</li> <li>New comparison examples: <ul> <li><code>examples/ab-compare.ts</code> and <code>examples/k6/ab-compare.ts</code> (closed model)</li> <li><code>examples/ab-compare-rps.ts</code> and <code>examples/k6/ab-compare-rps.ts</code> (fixed RPS)</li> </ul> </li> <li>A one-command comparison helper: <ul> <li><code>scripts/ab_compare_report.sh</code> (prints req/s, avg, p99, fail rate, H2 check)</li> </ul> </li> </ul> <h3 id="important-fairness-fix">Important fairness fix<a class="zola-anchor" href="#important-fairness-fix" aria-label="Anchor link for: important-fairness-fix">🔗</a></h3> <p>One easy mistake is RPS interpretation:</p> <ul> <li>In loadgen-rs, <code>--rps</code> is <strong>per client</strong></li> <li>Aggregate target is <code>rps_per_client * clients</code></li> </ul> <p>The Deno comparison script now maps a total target RPS to per-client RPS automatically (<code>RPS / CLIENTS</code>) to match k6 constant-arrival tests.</p> <h3 id="why-latency-can-differ-even-at-similar-req-s">Why latency can differ even at similar req/s<a class="zola-anchor" href="#why-latency-can-differ-even-at-similar-req-s" aria-label="Anchor link for: why-latency-can-differ-even-at-similar-req-s">🔗</a></h3> <p>Even when request rate is aligned, latency can diverge if concurrency shape is not aligned:</p> <ul> <li>number of active clients/VUs</li> <li>streams per connection (<code>-m</code> / <code>max_streams</code>)</li> <li>thread count and connection reuse behavior</li> </ul> <p>In short: equal target RPS does not automatically imply equal queueing behavior.</p> <h3 id="stage-profile-support-k6-like">Stage profile support (k6-like)<a class="zola-anchor" href="#stage-profile-support-k6-like" aria-label="Anchor link for: stage-profile-support-k6-like">🔗</a></h3> <p>I also added <code>examples/ab-stages.ts</code>, which emulates stage ramps by splitting stages into short segments and running sequential FFI runs. This gives a practical equivalent for basic k6 stage workflows.</p> <h3 id="correlation-path">Correlation path<a class="zola-anchor" href="#correlation-path" aria-label="Anchor link for: correlation-path">🔗</a></h3> <p>For dynamic data and sequence scenarios, <code>runScriptScenario()</code> supports:</p> <ul> <li><code>execution_mode: "fetch"</code> (pure Deno)</li> <li><code>execution_mode: "ffi-step"</code> (native step transport)</li> </ul> <p>The step API design and contract are documented in:</p> <ul> <li><code>docs/script-mode-ffi.md</code></li> <li><code>docs/ffi-step-api.md</code></li> <li><code>docs/ffi-contract.md</code></li> </ul> <h3 id="k6-parity-snapshot-as-of-march-1-2026">k6 Parity Snapshot (as of March 1, 2026)<a class="zola-anchor" href="#k6-parity-snapshot-as-of-march-1-2026" aria-label="Anchor link for: k6-parity-snapshot-as-of-march-1-2026">🔗</a></h3> <p>Short answer: Deno + loadgen-rs is now close to k6 for HTTP-focused scripted workflows, but it is not yet a full 1:1 replacement.</p> <p>What is now very similar:</p> <ul> <li>Scripted scenarios with step chains, checks, and extractors</li> <li>Correlation and dynamic data (<code>{{var}}</code> templates, JSON/regex/header/DOM extract)</li> <li>Cookie handling and redirect policies</li> <li>Two execution paths: pure <code>fetch</code> and native <code>ffi-step</code></li> <li>Multi-protocol support in native step mode (<code>h1</code>, <code>h2</code>, <code>h3</code>)</li> <li>k6-like terminal summary output and side-by-side comparison scripts</li> </ul> <p>Where k6 is still stronger:</p> <ul> <li>More mature executor/scenario system overall (arrival/vu/stage/ramping breadth)</li> <li>Larger built-in ecosystem (<code>k6/http</code>, ws/browser modules, extension landscape)</li> <li>Richer output/integration stack (cloud and common observability backends)</li> <li>Longer production track record for large shared test suites and CI pipelines</li> </ul> <p>Practical takeaway:</p> <p>If your priority is HTTP load + correlation + native H1/H2/H3 coverage with comparable CLI summaries, Deno + loadgen-rs is already highly usable. If you need the full k6 orchestration and ecosystem depth, k6 still has an edge.</p> <p>The distributed mode isn’t limited to raw throughput benchmarks. Workers also support <code>POST /run-script</code> — the same scripted scenario engine used locally, but executed remotely on each worker. This means you can run multi-step request sequences with checks, extractors, cookies, and response validation across a fleet of machines.</p> <p>The implementation required minimal changes: the worker-agent imports <code>runScriptScenario()</code> from the existing <code>script_mode.ts</code> and exposes it as a new HTTP endpoint. The controller sends a <code>ScriptScenarioConfig</code> (same type used for local script mode) to each worker and collects <code>ScriptRunResult</code> objects with per-check pass/fail counters.</p> <p>A practical example ships as <code>examples/html-check.ts</code> — a verification script that tests whether each worker can reach a target URL and parse the response correctly:</p> <pre class="giallo z-code"><code data-lang="typescript"><span class="giallo-l"><span class="z-storage z-type z-storage">const</span><span class="z-variable z-other z-constant z-variable"> config</span><span class="z-keyword z-keyword z-operator">:</span><span class="z-entity z-name"> ScriptScenarioConfig</span><span class="z-keyword z-keyword z-operator"> =</span><span> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> vus</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 3</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> duration_s</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 5</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> execution_mode</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">ffi-step</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> step_session_config</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> protocol</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">h2</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> insecure</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> response_headers</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> response_body_limit</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 2_000_000</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> steps</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-meta z-object z-member">{</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> name</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">html-get</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> method</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">GET</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> url</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-variable z-other z-variable"> targetUrl</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> capture_body</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> checks</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> status_in</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-constant z-constant z-numeric">200</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span><span class="z-constant z-constant z-numeric"> 403</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> body_includes</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-string">&quot;</span><span class="z-string">&lt;title&gt;Example&lt;/title&gt;</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> header_includes</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span><span class="z-string"> &quot;</span><span class="z-string">set-cookie</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">session=</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member"> }</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-meta z-object z-member">]</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span>}</span><span class="z-punctuation z-terminator">;</span></span></code></pre> <p>The controller sends this config to each worker via <code>POST /run-script</code>, then aggregates the results into a per-worker table. If the target returns both the HTML element and the cookie on every single response, the iteration count, HTML match count, and header match count should be identical — any mismatch indicates a parsing or check evaluation bug:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>=== per-worker results ===</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> worker iterations html_match header_match ok?</span></span> <span class="giallo-l"><span>-------------------------------------------------------------------------------------</span></span> <span class="giallo-l"><span> http://worker1:9091 62 62 62 PASS</span></span> <span class="giallo-l"><span> http://worker2:9091 47 47 47 PASS</span></span> <span class="giallo-l"><span>-------------------------------------------------------------------------------------</span></span> <span class="giallo-l"><span> TOTAL 109 109 109 PASS</span></span></code></pre> <p>All counts match perfectly across both workers. This validates the full chain: Terraform-provisioned infrastructure, Ansible-deployed workers, <code>ffi-step</code> execution mode with reqwest on the Rust side, response header capture, and the check evaluation engine — all working end-to-end on remote machines.</p> <p>The <code>header_includes</code> check type was added specifically for this use case. Unlike <code>header_exists</code> (which only checks presence) or <code>header_equals</code> (which requires an exact match), <code>header_includes</code> tests whether a header value <em>contains</em> a substring — essential for <code>Set-Cookie</code> headers where the full value includes expiry, path, and other attributes beyond the cookie name.</p> <h4 id="multi-step-login-logout-workflow">Multi-Step Login/Logout Workflow<a class="zola-anchor" href="#multi-step-login-logout-workflow" aria-label="Anchor link for: multi-step-login-logout-workflow">🔗</a></h4> <p>The scripted scenario engine really shines when steps depend on each other. A second distributed example — <code>examples/roundcube-login.ts</code> — verifies a full Roundcube Webmail login/logout cycle across workers. Unlike the single-step HTML check, this scenario chains three HTTP requests with state flowing between them:</p> <pre class="giallo z-code"><code data-lang="typescript"><span class="giallo-l"><span class="z-entity z-name">steps</span><span class="z-punctuation z-separator">:</span><span> [</span></span> <span class="giallo-l"><span> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> name</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">login-page</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> method</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">GET</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> url</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">https://mail.example.com/mail/</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> capture_body</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> checks</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> status_in</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-constant z-constant z-numeric">200</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> body_includes</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-string">&quot;</span><span class="z-string">Roundcube Webmail :: Welcome to Roundcube Webmail</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> extract</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-meta z-object z-member">{</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> type</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">dom</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> selector</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &#39;</span><span class="z-string">input[name=&quot;_token&quot;]</span><span class="z-string">&#39;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> attribute</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">value</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> as</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">token</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-meta z-object z-member">]</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> name</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">login-submit</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> method</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">POST</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> url</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">https://mail.example.com/mail/?_task=login</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> headers</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span><span class="z-string"> &quot;</span><span class="z-string">content-type</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">application/x-www-form-urlencoded</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> body</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">_task=login&amp;_action=login&amp;_timezone=...&amp;_token={{token}}&amp;_user=...&amp;_pass=...</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> capture_body</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> checks</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> status_in</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-constant z-constant z-numeric">200</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> body_includes</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-string">&quot;</span><span class="z-string">Roundcube Webmail :: Inbox</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span><span class="z-string"> &#39;</span><span class="z-string">data-label-msg=&quot;The list is empty.&quot;</span><span class="z-string">&#39;</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> extract</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-meta z-object z-member">{</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> type</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">regex</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> pattern</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &#39;</span><span class="z-string">&quot;request_token&quot;:&quot;([^&quot;]+)&quot;</span><span class="z-string">&#39;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> group</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> as</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">logout_token</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-meta z-object z-member">]</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> name</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">logout</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> method</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">GET</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> url</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-string"> &quot;</span><span class="z-string">https://mail.example.com/mail/?_task=logout&amp;_token={{logout_token}}</span><span class="z-string">&quot;</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> capture_body</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-constant z-constant z-language"> true</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> checks</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> {</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> status_in</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-constant z-constant z-numeric">200</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> body_includes</span><span class="z-meta z-object z-member z-punctuation z-separator">:</span><span class="z-meta z-object z-member"> [</span><span class="z-string">&quot;</span><span class="z-string">Roundcube Webmail :: Welcome to Roundcube Webmail</span><span class="z-string">&quot;</span><span class="z-meta z-object z-member">]</span><span class="z-meta z-object z-member z-punctuation z-separator">,</span></span> <span class="giallo-l"><span class="z-meta z-object z-member"> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span> }</span><span class="z-punctuation z-separator">,</span></span> <span class="giallo-l"><span>]</span></span></code></pre> <p>Each step depends on extracted values from the previous one: <code>{{token}}</code> (the CSRF token extracted from the login form’s hidden input via a DOM CSS selector — <code>input[name="_token"]</code>) flows into the POST body, and <code>{{logout_token}}</code> (the <code>request_token</code> extracted from the Roundcube JavaScript via regex) flows into the logout URL. The <code>dom</code> extractor parses the HTML response with deno-dom’s <code>DOMParser</code> and runs <code>querySelector</code> — far more robust than regex for structured HTML. Session cookies are managed automatically by the native <code>cookie_jar</code> — the <code>roundcube_sessid</code> and <code>roundcube_sessauth</code> cookies persist across all three steps within an iteration, and Roundcube’s logout response invalidates the session so the next iteration starts clean.</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>=== per-worker results ===</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span> worker iterations login_ok logout_ok ok?</span></span> <span class="giallo-l"><span>----------------------------------------------------------------------------------</span></span> <span class="giallo-l"><span> http://worker1:9091 45 45 45 PASS</span></span> <span class="giallo-l"><span> http://worker2:9091 45 45 45 PASS</span></span> <span class="giallo-l"><span>----------------------------------------------------------------------------------</span></span> <span class="giallo-l"><span> TOTAL 90 90 90 PASS</span></span></code></pre> <p>All 90 iterations completed successfully across both workers: every login extracted a valid CSRF token, every POST received the inbox page, and every logout returned to the login screen. This validates the full extractor → template substitution → cookie management pipeline running inside the Rust FFI layer on remote machines.</p> <h2 id="deep-dive-performance-analysis">Deep Dive: Performance Analysis<a class="zola-anchor" href="#deep-dive-performance-analysis" aria-label="Anchor link for: deep-dive-performance-analysis">🔗</a></h2> <p>To understand the performance gaps, I profiled loadgen-rs using <code>perf</code> and <a rel="noopener external" target="_blank" href="https://github.com/mstange/samply">samply</a>. The setup:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Build with debug symbols but full optimization</span></span> <span class="giallo-l"><span class="z-entity z-name">cargo</span><span class="z-string"> build</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-profile</span><span class="z-string"> profiling</span><span class="z-punctuation z-definition z-comment z-comment"> #</span><span class="z-comment"> inherits release, adds debug=2</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Record CPU profile</span></span> <span class="giallo-l"><span class="z-entity z-name">perf</span><span class="z-string"> record</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">g</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-call-graph</span><span class="z-string"> dwarf</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">o</span><span class="z-string"> /tmp/perf-h2.data</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> ./target/profiling/loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list</span><span class="z-string"> h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 64</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-insecure</span><span class="z-string"> https://127.0.0.1:8082/?s=256k</span></span></code></pre><h3 id="h2-cpu-profile">H2 CPU Profile<a class="zola-anchor" href="#h2-cpu-profile" aria-label="Anchor link for: h2-cpu-profile">🔗</a></h3> <table><thead><tr><th>Self %</th><th>Function</th><th>Category</th></tr></thead><tbody> <tr><td><strong>27%</strong></td><td><code>ring::aes_gcm_dec_update</code></td><td>TLS decrypt (AES-256-GCM)</td></tr> <tr><td><strong>21%</strong></td><td><code>__memmove_avx</code> (libc)</td><td>Memory copies</td></tr> <tr><td><strong>16%</strong></td><td>Kernel</td><td>TCP read/write syscalls</td></tr> <tr><td>2.0%</td><td><code>rustls::read_opaque_message_header</code></td><td>TLS framing</td></tr> <tr><td>1.4%</td><td><code>_mi_page_malloc</code></td><td>Allocator (mimalloc)</td></tr> <tr><td>1.3%</td><td><code>h2::Connection::poll2</code></td><td>H2 protocol state machine</td></tr> <tr><td>1.3%</td><td><code>h2::DynConnection::recv_frame</code></td><td>H2 frame processing</td></tr> <tr><td>1.2%</td><td><code>BytesMut::split_to</code></td><td>Buffer management</td></tr> </tbody></table> <p><strong>~64% of CPU time is spent outside of our code entirely</strong> — in TLS decryption, memory copies, and kernel syscalls. Only ~2.5% is in the h2 crate’s protocol logic.</p> <p>The H2 latency gap (1.8x vs h2load) is <em>not</em> a CPU bottleneck — it’s architectural. The h2 crate uses an async mpsc channel between <code>SendRequest</code> (our task) and the connection driver task (h2’s internal I/O loop). Each request crosses this channel boundary at least twice (send + receive), adding idle time that doesn’t show up in CPU profiles.</p> <p>h2load, by contrast, runs everything in a single libev event loop tick — nghttp2 processes the request, writes the frame, and reads the response without any task switching.</p> <h3 id="h3-cpu-profile">H3 CPU Profile<a class="zola-anchor" href="#h3-cpu-profile" aria-label="Anchor link for: h3-cpu-profile">🔗</a></h3> <table><thead><tr><th>Self %</th><th>Function</th><th>Category</th></tr></thead><tbody> <tr><td><strong>15%</strong></td><td><code>ring::aes_gcm_dec_update</code></td><td>TLS decrypt</td></tr> <tr><td><strong>13%</strong></td><td><code>__memmove_avx</code> (libc)</td><td>Memory copies</td></tr> <tr><td><strong>12%</strong></td><td>Kernel</td><td>UDP recvmsg syscalls</td></tr> <tr><td><strong>3.5%</strong></td><td><code>quinn_proto::Connection::process_payload</code></td><td>QUIC packet processing</td></tr> <tr><td><strong>3.1%</strong></td><td><code>quinn_proto::Connection::handle_decode</code></td><td>QUIC packet decode</td></tr> <tr><td><strong>2.9%</strong></td><td><code>quinn::endpoint::RecvState::poll_socket</code></td><td>UDP socket I/O</td></tr> <tr><td><strong>2.8%</strong></td><td><code>binary_heap::PeekMut::pop</code></td><td>Timer/scheduling (loss detection, CC)</td></tr> <tr><td>1.8%</td><td><code>ring::aes_gcm::open</code></td><td>Additional TLS overhead</td></tr> <tr><td>1.7%</td><td><code>quinn::RecvStream::poll_read_chunk</code></td><td>Stream receive</td></tr> <tr><td>1.7%</td><td><code>quinn_proto::Endpoint::handle</code></td><td>Packet routing/demux</td></tr> <tr><td>1.6%</td><td><code>h3::proto::varint::VarInt::decode</code></td><td>H3 framing</td></tr> </tbody></table> <p>The QUIC protocol stack (quinn-proto) accounts for <strong>~15% of CPU</strong> — compared to ~2.5% for h2. This is the fundamental cost of implementing QUIC in userspace Rust:</p> <ul> <li><strong>Per-packet crypto</strong>: QUIC encrypts each UDP datagram individually (vs TLS record batching over TCP)</li> <li><strong>Congestion control state machine</strong>: Quinn maintains timer heaps (<code>binary_heap::pop</code> at 2.8%) for loss detection and CC that TCP offloads to the kernel</li> <li><strong>Packet demux</strong>: <code>Endpoint::handle</code> routes each incoming UDP datagram to the correct connection — work that the kernel’s TCP stack does for free</li> </ul> <p>h2load uses ngtcp2 (C) and nghttp3 (C) which have tighter inner loops, less allocation overhead, and direct <code>writev()</code> for packet batching.</p> <h3 id="what-the-profile-means">What the Profile Means<a class="zola-anchor" href="#what-the-profile-means" aria-label="Anchor link for: what-the-profile-means">🔗</a></h3> <p>The profiling data tells a clear story: <strong>we’ve hit the performance ceiling of the Rust crate ecosystem</strong>, not a bug in our code.</p> <p>For H2, the bottleneck is TLS + memcpy + kernel — things we can’t optimize without replacing ring/rustls. For H3, quinn’s QUIC protocol processing adds ~15% overhead that ngtcp2 avoids by being a leaner C implementation.</p> <h3 id="optimization-attempts-what-worked-and-what-didn-t">Optimization Attempts (What Worked and What Didn’t)<a class="zola-anchor" href="#optimization-attempts-what-worked-and-what-didn-t" aria-label="Anchor link for: optimization-attempts-what-worked-and-what-didn-t">🔗</a></h3> <p>During development, I tried several optimizations to close the gap with h2load:</p> <h4 id="what-worked">What Worked<a class="zola-anchor" href="#what-worked" aria-label="Anchor link for: what-worked">🔗</a></h4> <table><thead><tr><th>Optimization</th><th>Effect</th><th>Protocol</th></tr></thead><tbody> <tr><td>Raw h2 crate instead of hyper</td><td>Eliminated one abstraction layer</td><td>H2</td></tr> <tr><td>Raw TCP + httparse instead of hyper for H1</td><td>+15-25% RPS vs hyper-based H1</td><td>H1</td></tr> <tr><td><code>initial_rtt(1ms)</code> for quinn</td><td>Faster congestion control ramp-up on low-latency links</td><td>H3</td></tr> <tr><td>1 GB stream/connection windows</td><td>Matches h2load’s window sizes</td><td>H2, H3</td></tr> <tr><td>H2 flow control: 512 KB batch release</td><td>Reduces WINDOW_UPDATE frame overhead</td><td>H2</td></tr> <tr><td>UDP socket buffers 4 MB via socket2</td><td>Fewer kernel drops under load</td><td>H3</td></tr> <tr><td>mimalloc global allocator</td><td>Better multi-threaded malloc performance</td><td>All</td></tr> <tr><td>Pre-built request templates</td><td>Avoid re-building HTTP requests per iteration</td><td>All</td></tr> <tr><td>Per-worker current-thread runtimes</td><td>No cross-thread lock contention</td><td>All</td></tr> </tbody></table> <h4 id="what-didn-t-work">What Didn’t Work<a class="zola-anchor" href="#what-didn-t-work" aria-label="Anchor link for: what-didn-t-work">🔗</a></h4> <table><thead><tr><th>Optimization</th><th>Expected</th><th>Actual</th><th>Why</th></tr></thead><tbody> <tr><td>Thread-local shared quinn::Endpoint</td><td>Less per-connection overhead</td><td><strong>-12% RPS</strong></td><td>UDP socket contention — all connections on one worker competed for the same socket</td></tr> <tr><td>H2 immediate flow control release (per-chunk)</td><td>Lower latency</td><td><strong>-5.6% RPS</strong></td><td>Generated too many WINDOW_UPDATE frames and cross-task messages</td></tr> <tr><td>Cubic congestion controller (instead of NewReno)</td><td>Match h2load’s ngtcp2 CC</td><td><strong>-7% RPS</strong></td><td>Quinn’s Cubic implementation doesn’t scale well with 512 concurrent connections on localhost</td></tr> <tr><td>http2 crate (0x676e67 fork with parking_lot)</td><td>Faster mutex in h2 hot path</td><td><strong>-2.3% RPS</strong></td><td>The h2 protocol logic is only 2.5% of CPU — faster locks don’t help when locks aren’t the bottleneck</td></tr> </tbody></table> <p>The failed optimizations were valuable lessons: profiling <em>before</em> optimizing would have saved time. The http2 fork experiment, for instance, was motivated by the assumption that h2’s internal locking was a bottleneck — but perf showed it was only 1.3% of CPU time.</p> <h3 id="architectural-comparison-loadgen-rs-vs-h2load">Architectural Comparison: loadgen-rs vs h2load<a class="zola-anchor" href="#architectural-comparison-loadgen-rs-vs-h2load" aria-label="Anchor link for: architectural-comparison-loadgen-rs-vs-h2load">🔗</a></h3> <p>Understanding <em>why</em> the gaps exist requires comparing the architectures:</p> <h4 id="h2-async-tasks-vs-single-event-loop">H2: Async Tasks vs Single Event Loop<a class="zola-anchor" href="#h2-async-tasks-vs-single-event-loop" aria-label="Anchor link for: h2-async-tasks-vs-single-event-loop">🔗</a></h4> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>loadgen-rs: Worker Task ──mpsc──▶ h2 Connection Driver Task ──▶ TCP</span></span> <span class="giallo-l"><span>h2load: libev event loop ──▶ nghttp2 ──▶ writev() ──▶ TCP</span></span></code></pre> <p>The h2 crate splits request handling across two async tasks connected by an mpsc channel. Each request crosses this boundary twice. h2load runs everything synchronously in one event loop tick — zero context switches per request.</p> <p>This explains the latency gap (1.8x) without a throughput gap: the pipeline stays full, but each individual request takes longer to complete.</p> <h4 id="h3-userspace-quic-vs-lean-c-implementation">H3: Userspace QUIC vs Lean C Implementation<a class="zola-anchor" href="#h3-userspace-quic-vs-lean-c-implementation" aria-label="Anchor link for: h3-userspace-quic-vs-lean-c-implementation">🔗</a></h4> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>loadgen-rs: quinn (Rust) ──▶ ring (Rust/asm) ──▶ UDP socket</span></span> <span class="giallo-l"><span>h2load: ngtcp2 (C) ──▶ OpenSSL/BoringSSL (C/asm) ──▶ UDP socket</span></span></code></pre> <p>Both implement the same QUIC protocol, but ngtcp2 has advantages:</p> <ul> <li>Tighter C inner loops with less allocation</li> <li>Direct <code>sendmsg</code>/<code>recvmsg</code> batching</li> <li>Fewer abstraction layers between protocol logic and I/O</li> </ul> <p>Quinn adds Tokio task scheduling, channel-based I/O between endpoint and connection state, and Rust’s safety overhead (bounds checks, reference counting on <code>Bytes</code>).</p> <h2 id="deep-dive-internal-architecture">Deep Dive: Internal Architecture<a class="zola-anchor" href="#deep-dive-internal-architecture" aria-label="Anchor link for: deep-dive-internal-architecture">🔗</a></h2> <h3 id="enum-based-driver-dispatch">Enum-Based Driver Dispatch<a class="zola-anchor" href="#enum-based-driver-dispatch" aria-label="Anchor link for: enum-based-driver-dispatch">🔗</a></h3> <p>Instead of <code>dyn Trait</code> with async boxing, I use an enum:</p> <pre class="giallo z-code"><code data-lang="rust"><span class="giallo-l"><span class="z-keyword">pub</span><span class="z-storage z-type z-storage"> enum</span><span class="z-entity z-name"> Connection</span><span> {</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> H1Raw</span><span>(</span><span class="z-entity z-name">h1_raw</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name">H1RawConnection</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> H2</span><span>(</span><span class="z-entity z-name">h2</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name">H2Connection</span><span>)</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name z-function"> H3</span><span>(</span><span class="z-entity z-name">h3</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name">H3Connection</span><span>)</span><span>,</span></span> <span class="giallo-l"><span>}</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-keyword">impl</span><span class="z-entity z-name"> Connection</span><span> {</span></span> <span class="giallo-l"><span class="z-keyword"> pub</span><span class="z-keyword"> async</span><span class="z-keyword"> fn</span><span class="z-entity z-name z-function"> send_request</span><span>(</span><span class="z-keyword z-keyword z-operator">&amp;</span><span class="z-storage">mut</span><span class="z-variable z-language"> self</span><span>,</span><span class="z-variable z-other z-variable"> config</span><span class="z-keyword z-keyword z-operator">:</span><span class="z-keyword z-keyword z-operator"> &amp;</span><span class="z-entity z-name">RequestConfig</span><span>)</span></span> <span class="giallo-l"><span class="z-keyword z-keyword z-operator"> -&gt;</span><span class="z-entity z-name"> Result</span><span>&lt;</span><span class="z-entity z-name">RequestResult</span><span>,</span><span class="z-entity z-name"> RequestError</span><span>&gt;</span></span> <span class="giallo-l"><span> {</span></span> <span class="giallo-l"><span class="z-keyword"> match</span><span class="z-variable z-language"> self</span><span> {</span></span> <span class="giallo-l"><span class="z-entity z-name"> Connection</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name z-function">H1Raw</span><span>(</span><span class="z-variable z-other z-variable">c</span><span>)</span><span class="z-keyword z-keyword z-operator"> =&gt;</span><span class="z-variable z-other z-variable"> c</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-entity z-name z-function">send_request</span><span>(</span><span class="z-variable z-other z-variable">config</span><span>)</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-keyword">await</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name"> Connection</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name z-function">H2</span><span>(</span><span class="z-variable z-other z-variable">c</span><span>)</span><span class="z-keyword z-keyword z-operator"> =&gt;</span><span class="z-variable z-other z-variable"> c</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-entity z-name z-function">send_request</span><span>(</span><span class="z-variable z-other z-variable">config</span><span>)</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-keyword">await</span><span>,</span></span> <span class="giallo-l"><span class="z-entity z-name"> Connection</span><span class="z-keyword z-keyword z-operator">::</span><span class="z-entity z-name z-function">H3</span><span>(</span><span class="z-variable z-other z-variable">c</span><span>)</span><span class="z-keyword z-keyword z-operator"> =&gt;</span><span class="z-variable z-other z-variable"> c</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-entity z-name z-function">send_request</span><span>(</span><span class="z-variable z-other z-variable">config</span><span>)</span><span class="z-keyword z-keyword z-operator">.</span><span class="z-keyword">await</span><span>,</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span> }</span></span> <span class="giallo-l"><span>}</span></span></code></pre> <p>Benefits:</p> <ul> <li>No heap allocation per dispatch</li> <li>Compiler can inline</li> <li>No <code>Pin&lt;Box&lt;dyn Future&gt;&gt;</code> overhead</li> </ul> <h3 id="ffi-two-api-design-batch-vs-step">FFI Two-API Design (Batch vs Step)<a class="zola-anchor" href="#ffi-two-api-design-batch-vs-step" aria-label="Anchor link for: ffi-two-api-design-batch-vs-step">🔗</a></h3> <p>The Deno integration isn’t just a thin wrapper — it’s built on a Cargo workspace with a dedicated <code>loadgen-ffi</code> crate that compiles to a 6.4 MB shared library (<code>libloadgen_ffi.so</code>) exposing 13 C ABI functions via <code>Deno.dlopen()</code>.</p> <h4 id="two-distinct-ffi-apis">Two distinct FFI APIs<a class="zola-anchor" href="#two-distinct-ffi-apis" aria-label="Anchor link for: two-distinct-ffi-apis">🔗</a></h4> <p>The shared library provides two API levels, each serving a different use case:</p> <p><strong>1. Batch API</strong> — for high-throughput load generation:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>loadgen_create(config_json) → handle</span></span> <span class="giallo-l"><span>loadgen_run(handle) → report_json (nonblocking in Deno)</span></span> <span class="giallo-l"><span>loadgen_metrics_snapshot(handle) → json</span></span> <span class="giallo-l"><span>loadgen_destroy(handle)</span></span></code></pre> <p>This wraps the existing <code>run_from_config()</code> engine — the same optimized H1/H2/H3 drivers, per-worker runtimes, and HdrHistograms that match h2load performance. A single <code>loadgen_run</code> call spawns the full benchmark and returns a complete <code>RunReport</code> as JSON.</p> <p><strong>2. Step API</strong> — for scripted request sequences with correlation:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>loadgen_step_session_create(session_config_json) → session_handle</span></span> <span class="giallo-l"><span>loadgen_step_execute(session_handle, request_json) → response_json (nonblocking)</span></span> <span class="giallo-l"><span>loadgen_step_session_reset(session_handle)</span></span> <span class="giallo-l"><span>loadgen_step_session_destroy(session_handle)</span></span></code></pre> <p>The Step API uses <strong>reqwest</strong> as its HTTP client rather than the custom drivers. This is a deliberate architectural choice: reqwest provides built-in cookie jars, redirect following, H1/H2/H3 support, and connection pooling — features that the raw benchmark drivers intentionally omit for performance. In script mode, correctness (cookies, redirects, body capture) matters more than raw throughput.</p> <h4 id="panic-safe-ffi-boundary">Panic-safe FFI boundary<a class="zola-anchor" href="#panic-safe-ffi-boundary" aria-label="Anchor link for: panic-safe-ffi-boundary">🔗</a></h4> <p>Every exported function is wrapped in <code>std::panic::catch_unwind()</code> to prevent Rust panics from unwinding across the C ABI boundary (which would be undefined behavior). Errors are stored in a thread-safe slot (<code>OnceLock&lt;Mutex&lt;Option&lt;String&gt;&gt;&gt;</code>) and retrieved via <code>loadgen_last_error()</code>. All returned <code>char*</code> strings must be freed with <code>loadgen_free_string()</code> — the contract is documented in <code>docs/ffi-contract.md</code>.</p> <h4 id="typescript-script-engine">TypeScript script engine<a class="zola-anchor" href="#typescript-script-engine" aria-label="Anchor link for: typescript-script-engine">🔗</a></h4> <p>On the Deno side, <code>runScriptScenario()</code> in <code>ts/script_mode.ts</code> implements a k6-inspired scenario runner with:</p> <ul> <li><strong>VU parallelism</strong>: N virtual users run as concurrent <code>Promise.all</code> tasks</li> <li><strong>Duration-based execution</strong>: VUs loop until the target duration expires</li> <li><strong>Two execution modes</strong>: <code>fetch</code> (Deno’s native HTTP client) and <code>ffi-step</code> (Rust reqwest via the Step API)</li> <li><strong>Template engine</strong>: <code>{{variable}}</code> substitution in URLs, headers, and bodies</li> <li><strong>Extractor system</strong>: Capture values from JSON paths, response headers, regex groups, or DOM CSS selectors (via deno-dom) — stored in per-VU state for use in subsequent steps</li> <li><strong>Check system</strong>: Eight check types including <code>status_in</code>, <code>body_includes</code>, <code>header_exists</code>, <code>header_equals</code>, <code>header_includes</code>, <code>json_path_exists</code>, <code>json_path_equals</code>, and <code>regex_match</code></li> <li><strong>Cookie jar</strong>: Full RFC-aware implementation in TypeScript (for <code>fetch</code> mode) with domain/path matching, expiry handling, and Secure attribute support</li> </ul> <p>The <code>ffi-step</code> mode delegates cookie management and redirects to reqwest on the Rust side, while the <code>fetch</code> mode handles everything in TypeScript. Both modes share the same check and extractor evaluation logic.</p> <p>The full scripting documentation is in <a rel="noopener external" target="_blank" href="https://github.com/git001/loadgen-rs/blob/main/docs/loadgen-script.md">loadgen-script</a></p> <h4 id="example-coverage">Example coverage<a class="zola-anchor" href="#example-coverage" aria-label="Anchor link for: example-coverage">🔗</a></h4> <p>The <code>examples/</code> directory in <a rel="noopener external" target="_blank" href="https://github.com/git001/loadgen-rs/">loadgen-rs repository</a> demonstrates the full feature surface:</p> <table><thead><tr><th>Example</th><th>What it demonstrates</th></tr></thead><tbody> <tr><td><code>simple.ts</code></td><td>Minimal FFI benchmark (10 requests, single client)</td></tr> <tr><td><code>correlation_mvp.ts</code></td><td>Login, extract JWT token, use in follow-up request</td></tr> <tr><td><code>cookie_redirect_local_mvp.ts</code></td><td>Embedded Deno server with Set-Cookie + 302 redirect chain</td></tr> <tr><td><code>protocol-matrix.ts</code></td><td>H1/H2/H3 batch + script mode for all three protocols</td></tr> <tr><td><code>ab-stages.ts</code></td><td>Ramping VUs with stage interpolation (k6-like stages)</td></tr> <tr><td><code>native_step_skeleton.ts</code></td><td>Direct Step API usage without the scenario runner</td></tr> <tr><td><code>all-functionality.ts</code></td><td>Full integration test suite using Chai assertions</td></tr> </tbody></table> <p>The <code>cookie_redirect_local_mvp.ts</code> example is particularly interesting: it spins up a local Deno HTTP server that sets cookies and issues redirects, then runs the script scenario against it — a self-contained integration test that validates the entire cookie + redirect + check pipeline.</p> <h2 id="appendix-commands-and-reproducibility">Appendix: Commands and Reproducibility<a class="zola-anchor" href="#appendix-commands-and-reproducibility" aria-label="Anchor link for: appendix-commands-and-reproducibility">🔗</a></h2> <h3 id="equivalent-command-pairs-h2load-vs-loadgen-rs">Equivalent Command Pairs (h2load vs loadgen-rs)<a class="zola-anchor" href="#equivalent-command-pairs-h2load-vs-loadgen-rs" aria-label="Anchor link for: equivalent-command-pairs-h2load-vs-loadgen-rs">🔗</a></h3> <p>To compare both tools fairly, use equivalent flags and remember: <code>--rps</code> is <strong>per client</strong> in both tools.</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1) H1 count mode (same request budget)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 200</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 200</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2) H1 duration mode with pacing (~200 aggregate rps: 10 clients * 20 rps/client)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-h1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3) H2 duration mode with pacing (same effective target)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list</span><span class="z-string"> h2</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4) H3 duration mode with pacing (same effective target)</span></span> <span class="giallo-l"><span class="z-entity z-name">h2load</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h3</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span> <span class="giallo-l"><span class="z-entity z-name">loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list</span><span class="z-string"> h3</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-duration</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">n</span><span class="z-constant z-constant z-numeric"> 0</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 10</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">m</span><span class="z-constant z-constant z-numeric"> 1</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rps</span><span class="z-constant z-constant z-numeric"> 20</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.local:8082/?s=256k</span><span class="z-string">&quot;</span></span></code></pre><h3 id="cli-compatibility-map">CLI Compatibility Map<a class="zola-anchor" href="#cli-compatibility-map" aria-label="Anchor link for: cli-compatibility-map">🔗</a></h3> <p>The table below maps <code>loadgen-rs</code> options to their closest <code>h2load</code> equivalent.</p> <table><thead><tr><th><code>loadgen-rs</code></th><th><code>h2load</code></th><th>Mapping</th><th>Notes</th></tr></thead><tbody> <tr><td><code>&lt;URL&gt;</code></td><td><code>&lt;URI&gt;...</code></td><td>partial</td><td><code>h2load</code> accepts multiple URIs</td></tr> <tr><td><code>-n &lt;REQUESTS&gt;</code></td><td><code>-n, --requests=&lt;N&gt;</code></td><td>same</td><td></td></tr> <tr><td><code>-D, --duration</code></td><td><code>-D, --duration</code></td><td>same</td><td></td></tr> <tr><td><code>--warm-up-time</code></td><td><code>--warm-up-time</code></td><td>same</td><td></td></tr> <tr><td><code>--ramp-up-time</code></td><td><code>none</code></td><td>no direct equivalent</td><td>Closest knobs are <code>-r/--rate</code> + <code>--rate-period</code>, but semantics differ</td></tr> <tr><td><code>-c &lt;CLIENTS&gt;</code></td><td><code>-c, --clients=&lt;N&gt;</code></td><td>same</td><td></td></tr> <tr><td><code>-t &lt;THREADS&gt;</code></td><td><code>-t, --threads=&lt;N&gt;</code></td><td>same</td><td></td></tr> <tr><td><code>-m &lt;MAX_STREAMS&gt;</code></td><td><code>-m, --max-concurrent-streams=&lt;N&gt;</code></td><td>same</td><td></td></tr> <tr><td><code>--h1</code></td><td><code>--h1</code></td><td>same</td><td></td></tr> <tr><td><code>--alpn-list h2|h3</code></td><td><code>--alpn-list=&lt;LIST&gt;</code></td><td>partial</td><td><code>h2load</code> allows arbitrary ALPN lists</td></tr> <tr><td><code>--connect-timeout</code></td><td><code>none</code></td><td>no direct equivalent</td><td>Only related connection-level timeout controls exist (<code>-T</code>, <code>-N</code>)</td></tr> <tr><td><code>--request-timeout</code></td><td><code>none</code></td><td>no direct equivalent</td><td>No per-request timeout flag in <code>h2load</code></td></tr> <tr><td><code>--tcp-quickack</code></td><td><code>none</code></td><td>none</td><td></td></tr> <tr><td><code>--method &lt;METHOD&gt;</code></td><td><code>none</code></td><td>no direct equivalent</td><td><code>h2load</code> is primarily GET; <code>POST</code> is enabled via <code>-d</code></td></tr> <tr><td><code>-H, --header</code></td><td><code>-H, --header</code></td><td>same</td><td></td></tr> <tr><td><code>-d, --data &lt;DATA&gt;</code></td><td><code>-d, --data=&lt;PATH&gt;</code></td><td>partial</td><td><code>loadgen-rs</code>: inline body; <code>h2load</code>: file path</td></tr> <tr><td><code>--data-file &lt;PATH&gt;</code></td><td><code>-d, --data=&lt;PATH&gt;</code></td><td>close equivalent</td><td></td></tr> <tr><td><code>--rps &lt;RPS&gt;</code></td><td><code>--rps=&lt;N&gt;</code></td><td>same</td><td>per-client target in both tools (aggregate <code>RPS * clients</code>)</td></tr> <tr><td><code>-k, --insecure</code></td><td><code>none</code></td><td>no CLI equivalent</td><td>No direct insecure toggle in <code>h2load</code> CLI</td></tr> <tr><td><code>--tls-ciphers</code></td><td><code>--ciphers</code>, <code>--tls13-ciphers</code></td><td>partial</td><td><code>h2load</code> splits TLS &lt;=1.2 and TLS 1.3 ciphers</td></tr> <tr><td><code>--tls-ca &lt;PATH&gt;</code></td><td><code>none</code></td><td>none</td><td></td></tr> <tr><td><code>-o, --output &lt;PATH&gt;</code></td><td><code>none</code></td><td>none</td><td><code>h2load</code> offers <code>--log-file</code> (per-request TSV), not run-level JSON/CSV</td></tr> <tr><td><code>--format jsonl|csv</code></td><td><code>none</code></td><td>none</td><td></td></tr> <tr><td><code>--no-human</code></td><td><code>none</code></td><td>none</td><td></td></tr> <tr><td><code>--tail-friendly</code></td><td><code>none</code></td><td>none</td><td></td></tr> <tr><td><code>--metrics-sample</code></td><td><code>none</code></td><td>none</td><td></td></tr> </tbody></table> <h3 id="container-baseline">Container Baseline<a class="zola-anchor" href="#container-baseline" aria-label="Anchor link for: container-baseline">🔗</a></h3> <p>The multi-stage Containerfile builds the binary in a Rust image and copies it into a minimal Debian Slim image with <code>bash</code> and <code>curl</code>, with HTTP/3 support built in:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-entity z-name">podman</span><span class="z-string"> build</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-string"> loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">f</span><span class="z-string"> Containerfile</span><span class="z-string"> .</span></span> <span class="giallo-l"><span class="z-entity z-name">podman</span><span class="z-string"> run</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-rm</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-network</span><span class="z-string"> host</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-cap-add=SYS_ADMIN</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-security-opt</span><span class="z-string"> seccomp=unconfined</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-string"> loadgen-rs</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-string"> 10s</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 8</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">t</span><span class="z-constant z-constant z-numeric"> 4</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-alpn-list=h2</span><span class="z-string"> https://target:8443/</span></span></code></pre> <p><code>--network host</code> avoids NAT overhead. <code>--cap-add=SYS_ADMIN</code> enables io_uring inside the container.</p> <h3 id="test-server-setup">Test Server Setup<a class="zola-anchor" href="#test-server-setup" aria-label="Anchor link for: test-server-setup">🔗</a></h3> <p>The test server used in this post is <code>haterm</code> from <code>HAProxy</code>, created and started with the following commands:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span># create combined.pem in TLS-PATH</span></span> <span class="giallo-l"><span>cat server.crt ca.crt server.key &gt; combined.pem</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># build container image</span></span> <span class="giallo-l"><span>buildah bud -f Containerfile-hap-git -t hap-own</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span># run haterm</span></span> <span class="giallo-l"><span>podman run -d --rm --name hap-own \</span></span> <span class="giallo-l"><span> --network host \</span></span> <span class="giallo-l"><span> --entrypoint=/usr/local/sbin/haterm \</span></span> <span class="giallo-l"><span> -v &lt;TLS-PATH&gt;/tls:/mnt:ro \</span></span> <span class="giallo-l"><span> localhost/haterm:latest \</span></span> <span class="giallo-l"><span> -L 127.0.0.1:8081 \</span></span> <span class="giallo-l"><span> -F &#39;bind quic4@127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h3&#39; \</span></span> <span class="giallo-l"><span> -F &#39;bind 127.0.0.1:8082 ssl crt /mnt/combined.pem alpn h2&#39;</span></span></code></pre> <p>The used <a rel="noopener external" target="_blank" href="https://github.com/git001/loadgen-rs/blob/main/Containerfile-hap-git">Containerfile-hap-git</a></p> <h3 id="cloud-deployment-terraform-ansible">Cloud Deployment (Terraform + Ansible)<a class="zola-anchor" href="#cloud-deployment-terraform-ansible" aria-label="Anchor link for: cloud-deployment-terraform-ansible">🔗</a></h3> <p>Running distributed benchmarks on your local machines is useful for development, but real load testing demands dedicated cloud infrastructure — consistent hardware, dedicated NICs, and geographic proximity to the target. loadgen-rs ships with a complete Terraform + Ansible pipeline for Hetzner Cloud that takes you from zero to running workers in under five minutes.</p> <p>The Terraform configuration provisions dedicated-vCPU instances (CCX series), places them in a private network, and configures a firewall that allows SSH and the worker-agent port. A single <code>generate-inventory.sh</code> script bridges the gap between Terraform and Ansible — it reads the Terraform outputs and writes a ready-to-use <code>ansible/inventory.ini</code> with the public IPs:</p> <pre class="giallo z-code"><code data-lang="plain"><span class="giallo-l"><span>terraform apply</span></span> <span class="giallo-l"><span>│</span></span> <span class="giallo-l"><span>├── hcloud_server.worker[0] → 49.13.x.x (loadgen-worker-0)</span></span> <span class="giallo-l"><span>├── hcloud_server.worker[1] → 49.13.y.y (loadgen-worker-1)</span></span> <span class="giallo-l"><span>├── hcloud_network → 10.0.1.0/24 (private)</span></span> <span class="giallo-l"><span>└── hcloud_firewall → SSH + port 9091</span></span> <span class="giallo-l"><span>│</span></span> <span class="giallo-l"><span>▼</span></span> <span class="giallo-l"><span>./generate-inventory.sh</span></span> <span class="giallo-l"><span>│</span></span> <span class="giallo-l"><span>▼</span></span> <span class="giallo-l"><span>ansible/inventory.ini</span></span> <span class="giallo-l"><span>├── loadgen-worker-0 ansible_host=49.13.x.x worker_port=9091</span></span> <span class="giallo-l"><span>└── loadgen-worker-1 ansible_host=49.13.y.y worker_port=9091</span></span> <span class="giallo-l"><span>│</span></span> <span class="giallo-l"><span>▼</span></span> <span class="giallo-l"><span>ansible-playbook deploy-workers.yml</span></span> <span class="giallo-l"><span>│</span></span> <span class="giallo-l"><span>▼</span></span> <span class="giallo-l"><span>Workers ready — POST /run from controller</span></span></code></pre> <p>The full workflow from provisioning to results:</p> <pre class="giallo z-code"><code data-lang="shellscript"><span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 1. Provision Hetzner Cloud machines</span></span> <span class="giallo-l"><span class="z-support z-support z-function">cd</span><span class="z-string"> terraform</span></span> <span class="giallo-l"><span class="z-entity z-name">cp</span><span class="z-string"> terraform.tfvars.example</span><span class="z-string"> terraform.tfvars</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> Set: hcloud_token, ssh_key_name, worker_count=4, worker_type=ccx33</span></span> <span class="giallo-l"><span class="z-entity z-name">terraform</span><span class="z-string"> init</span><span class="z-punctuation z-separator"> &amp;&amp;</span><span class="z-entity z-name"> terraform</span><span class="z-string"> apply</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 2. Generate Ansible inventory from Terraform state</span></span> <span class="giallo-l"><span class="z-entity z-name">./generate-inventory.sh</span></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> → ansible/inventory.ini with 4 workers</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 3. Deploy + benchmark + cleanup in one command</span></span> <span class="giallo-l"><span class="z-support z-support z-function">cd</span><span class="z-string"> ..</span></span> <span class="giallo-l"><span class="z-entity z-name">scripts/distributed-remote.sh</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">i</span><span class="z-string"> ansible/inventory.ini</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-target-url</span><span class="z-string"> &quot;</span><span class="z-string">https://bench.target:8443/?s=256k</span><span class="z-string">&quot;</span><span class="z-constant z-character"> \</span></span> <span class="giallo-l"><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">c</span><span class="z-constant z-constant z-numeric"> 1000</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">D</span><span class="z-constant z-constant z-numeric"> 60</span><span class="z-constant z-constant z-other"> -</span><span class="z-constant z-constant z-other">-stop-after</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span class="z-punctuation z-definition z-comment z-comment">#</span><span class="z-comment"> 4. Tear down when done</span></span> <span class="giallo-l"><span class="z-support z-support z-function">cd</span><span class="z-string"> terraform</span><span class="z-punctuation z-separator"> &amp;&amp;</span><span class="z-entity z-name"> terraform</span><span class="z-string"> destroy</span></span></code></pre> <p>The Ansible playbook (<code>deploy-workers.yml</code>) handles everything on the remote machines: installing Deno if not present, copying the FFI shared library and TypeScript files, deploying a systemd service unit (with a tmux fallback for systems without systemd), and running health checks to confirm each worker is ready. The matching <code>stop-workers.yml</code> playbook cleanly shuts everything down.</p> <p>This separation of concerns — Terraform owns infrastructure, Ansible owns configuration, the shell script orchestrates the workflow — means you can mix and match. Already have machines? Skip Terraform, write <code>inventory.ini</code> by hand. Prefer a different cloud? Replace the <code>.tf</code> files, keep the same Ansible playbooks and scripts. The worker-agent doesn’t care how it got deployed.</p> <table><thead><tr><th>Component</th><th>Responsibility</th></tr></thead><tbody> <tr><td><code>terraform/*.tf</code></td><td>Provision Hetzner VMs, network, firewall</td></tr> <tr><td><code>terraform/generate-inventory.sh</code></td><td>Bridge: Terraform outputs → Ansible inventory</td></tr> <tr><td><code>ansible/deploy-workers.yml</code></td><td>Install Deno, deploy files, start workers</td></tr> <tr><td><code>ansible/stop-workers.yml</code></td><td>Stop workers (systemd or tmux)</td></tr> <tr><td><code>scripts/distributed-remote.sh</code></td><td>End-to-end: deploy → benchmark → stop</td></tr> </tbody></table> <h2 id="conclusion">Conclusion<a class="zola-anchor" href="#conclusion" aria-label="Anchor link for: conclusion">🔗</a></h2> <p>loadgen-rs is a lean, focused tool: it performs exactly one run, outputs the results as JSONL, and leaves the orchestration (warmup, parameter sweeps, plotting) to the calling script. The combination of raw h2, quinn, and rustls covers all three HTTP protocols — and thanks to per-worker runtimes and a lock-free design, the client itself is not the bottleneck.</p> <p>CPU profiling confirmed this: over 60% of execution time is spent in TLS decryption, memory copies, and kernel syscalls — code paths shared by any HTTP benchmark client regardless of language. The remaining gaps to h2load are architectural properties of the Rust crate ecosystem (h2’s async channel design, quinn’s userspace QUIC overhead) rather than optimization opportunities in loadgen-rs itself.</p> <p>With the Deno FFI layer, distributed mode, and Terraform/Ansible pipeline, loadgen-rs now serves five usage patterns: shell scripts that need a fast, h2load-compatible CLI tool; TypeScript users who want k6-style scripted scenarios with native H1/H2/H3 performance; multi-machine deployments where a single process can’t generate enough load; cloud-provisioned benchmarks where you spin up dedicated Hetzner instances, run the test, and tear everything down in minutes; and distributed scripted scenarios where workers execute multi-step request sequences with checks and response validation on remote machines. The two-API design (batch for throughput, step for correlation) plus the controller/worker/infrastructure stack bridges all five without compromising any of them.</p> <h3 id="code-and-repository-link">Code and Repository Link<a class="zola-anchor" href="#code-and-repository-link" aria-label="Anchor link for: code-and-repository-link">🔗</a></h3> <p>The code is on <a rel="noopener external" target="_blank" href="https://github.com/git001/loadgen-rs">loadgen-rs - GitHub</a> — PRs welcome.</p>