Configuration

A suite is a YAML file, himorime.yaml by default. run, compare, ci, list and validate accept suite files or directories containing himorime.yaml and *.himorime.yaml files.

Editor support #

Put this comment on the first line for completion and validation in editors that support the YAML language server:

# yaml-language-server: $schema=https://raw.githubusercontent.com/nao1215/himorime/main/schema/himorime.schema.json

himorime embeds the same schema and validates it before running a suite.

himorime v0.4.0 keeps version: "1" but changes the layout and removes the old spellings. See GitHub Actions before copying it into a workflow.

For a runnable first suite, use Getting started. The Cookbook has examples for budgets, regression checks, fixtures and CI.

Migrating from the previous layout #

Commands #

Use a list to run a program directly: command: [jq, -c, ., input.json]. Use a string only with shell: true when shell syntax is required. Variables in shell strings are quoted by himorime and must not be quoted again. On Windows, substituted values cannot contain quotes, percent signs, line breaks or trailing backslashes. Shell startup is part of the measured time.

Runs #

runs fixes the sample count. Without it, himorime runs until min_runs and min_time are satisfied, up to max_runs; comparisons also require regression.min_samples. Commands and revisions are interleaved in a seeded order. --runs and --warmup override the suite for quick checks.

Metrics, budgets and tolerances #

Latency is always measured. metrics enables throughput, CPU time and peak RSS; Metrics defines their scope and platform support.

metrics:
  throughput:
    work: {value: 1000, unit: records}   # or {file_size: path}
  cpu: true
  memory: true
  unsupported: fail        # or skip

cpu, memory and unsupported may be defaults or benchmark overrides. Throughput belongs to one benchmark because it needs that benchmark’s work value or input file.

A budget is an absolute limit for one command, metric and aggregation. Available aggregations are min, max, mean, median and percentiles from p1 to p99.9.

commands:
  mytool:
    command: [mytool, input.json]
    budget:
      latency: {p95: "<= 30ms"}
      throughput: {median: ">= 50MiB/s"}
      cpu_total: {median: "<= 25ms"}
      peak_rss: {max: "<= 64MiB"}

Latency, CPU time and peak RSS use upper bounds; throughput uses lower bounds. A budget for an unmeasured metric is invalid.

regression configures latency, total CPU time and peak RSS comparisons. Throughput comparisons are derived from latency; use a throughput budget for an absolute rate limit.

regression:
  confidence: 0.95
  latency: {gate: false}                  # compared and reported, never fails
  cpu_total: {max_percent: 15, min_difference: 5ms}
  peak_rss: {max_percent: 5, min_difference: 1MiB}

statistic selects median or mean, max_percent sets the relative tolerance, min_difference ignores smaller absolute changes, and gate: false reports a verdict without changing the exit status. See Regression detection for the decision rules.

Hooks #

• setup runs once per benchmark and revision before warmup.
• prepare_each runs outside the measured time before every warmup and measured run.
• cleanup runs after the benchmark, including after failure or interruption.

Background processes are stopped when their command or hook exits, so hooks cannot host a resident service. A failed hook fails the benchmark or prepared command; its standard-error tail is included in the report.

Reports #

report.outputs writes reports after every run. Paths are relative to the suite file. A Markdown output with section updates matching himorime:begin and himorime:end markers instead of replacing the file. report.versions records the first non-empty output line from each version command.

report:
  outputs:
    - format: json
      path: bench/result.json
    - format: markdown
      path: docs/compare.md
      section: speed
  versions:
    jc: [jc, --version]
    jo: [jo, -v]

Version commands run once before the build, without a shell, with LC_ALL=C and a 30-second timeout. A failed command fails the suite. See Reports for formats and section updates.

Standard input and output #

stdin is a fixture path (stdin: testdata/input.txt), stdin: {file: ...}, or inline text (stdin: {content: "..."}). The file is reopened for every run, so every run reads it from the first byte.

A command’s standard output and standard error are discarded by default and never mixed with himorime’s own report. Set stdout or stderr to a path inside ${workdir} to keep the latest run’s output. When a run fails, the last lines of its standard error are shown in the report either way, with secrets masked.

Set terminal: true for an interactive shell, line editor or TUI. On Linux and macOS, himorime runs the command on an 80 by 24 pseudo-terminal and types stdin one key at a time:

- name: shell completion
  terminal: true
  stdin: {content: "SELECT * FROM us\t\n.exit\n"}
  commands:
    sqly:
      command: ["${artifact}", "${workdir}/users.csv"]

The input must end the program, for example with .exit, q or Ctrl-D (\u0004); otherwise timeout stops it. Key waits count toward latency. Terminal input and output are outside the measured process tree, and stderr cannot be captured separately.

Variables #

Commands, cwd, env values, stdin and hooks may use these variables:

An unknown variable is a validation error. ${artifact} without a build and ${workdir} inside the build are validation errors too.

Paths #

Every relative path of a suite is relative to ${root}, the directory holding the suite file inside the tree being measured. In a plain run that is the directory of the suite file. In a comparison it is that directory in the base revision’s temporary worktree for the base, and in your working tree for the head, so each revision runs its own code:

${head_root} names the working tree for both revisions. Use it when both revisions must read the same fixture:

benchmarks:
  - name: parse the large log
    stdin: "${head_root}/testdata/large.log"   # the same input for base and head
    commands:
      parser:
        command: [sh, parse.sh]                # each revision's own parse.sh

Absolute paths and ~ are rejected. Relative paths and resolved symbolic links must stay inside the repository, the base worktree or ${workdir}. Output and report paths cannot climb above their base directory. A suite that does not exist in the base revision is measured only in the working tree; see Regression detection.

Environment #

Commands and hooks inherit himorime’s environment. env from defaults, the benchmark and the command or hook is applied in that order. Reports omit the environment and show command lines before ${env:NAME} substitution.

Defaults #

A benchmark inherits defaults; a command inherits its benchmark. The most specific value wins.

Values and units #

Every value is typed; himorime never compares a formatted string.

A bare number is rejected where a unit is expected, because 10 could mean ten of anything. Units are case-sensitive: 64mib is an error. Durations may not exceed 24h; a budget’s limit and a work value must be greater than zero.

A budget is an operator followed by a value, such as "< 20ms", "<= 64MiB" or ">= 50MiB/s". Quote it: YAML reads a leading < fine, but a leading > starts a folded block.

Validation #

himorime validate checks the following without running commands:

1. YAML syntax and duplicate keys.
2. The schema: unknown keys, types, required keys, durations, budgets, percentages, names and path shapes. Every problem is listed at once.
3. Semantic rules, including unique names, valid references, run bounds, variable availability, metric directions and throughput units.

himorime compare and himorime ci additionally refuse a benchmark whose runs, max_runs or --runs is below regression.min_samples, because such a comparison could never be conclusive; adaptive runs of a comparison continue until min_samples.

Field reference #

Top level #

defaults #

build, setup, prepare_each, cleanup #

benchmarks[] #

benchmarks[].commands.NAME #

benchmarks[].commands.NAME.budget #

benchmarks[].metrics #

benchmarks[].metrics.throughput.work #

regression #

regression.latency, regression.cpu_total, regression.peak_rss #

report #

report.outputs[] #