Cookbook

Every recipe here is a suite under examples/ that you can run from a clone of the repository. The end-to-end suite runs each recipe (test/e2e/atago/cookbook.atago.yaml, one scenario per heading) on Linux, macOS and Windows, and a test fails if a recipe and its scenario drift apart. Recipes that reference other CLIs use git, which CI already has; nothing is downloaded.

The examples measure two small programs in examples/tools: wordcount, a word counter with a streaming, an in-memory and a parallel implementation, and sleepy, a stand-in whose run time, CPU use, child processes and exit status you control.

Keep a CLI within a budget #

Protect the startup latency of a CLI #

Protect the throughput of large CSV and JSON processing #

Catch regressions against a base revision #

Detect a CPU time regression #

Detect a peak RSS regression #

Compare the Git base and head on every metric #

Compare a script-based CLI without a build step #

Gate on CPU time and memory, report latency only #

Run it in CI #

Fail a GitHub Actions job when a performance budget is violated #

Cope with noise on GitHub-hosted runners #

Compare programs or inputs #

Compare similar CLIs on the same input #

Compare parsers on a stdin fixture #

Compare small, medium and large inputs #

Understand the geometric mean #

Understand the numbers #

Understand CPU utilization above 100% #

Understand process tree measurement on each OS #

Handle a metric this platform cannot measure #

Keep results #

Save JSON, CSV and Markdown reports locally #

Control state between runs #

Reset state before every run #

Clean up after success, failure or interruption #

Benchmark a warm cache #

Choose what runs #

Run only smoke benchmarks #

Handle failures #

Stop a command that hangs #

Treat a non-zero exit as an error #

Use a shell pipeline #