Compare small, medium and large inputs

Performance depends on input size, and you want each size as its own case instead of one averaged number, with fixtures generated rather than committed.

# yaml-language-server: $schema=../../schema/himorime.schema.json
#
# Recipes: separate cases per input size, fixtures generated by setup, and a
# geometric mean across cases.
# https://nao1215.github.io/himorime/cookbook/#compare-small-medium-and-large-inputs
version: "1"

name: input sizes
description: The same two implementations on small, medium and large inputs.

build:
  command: [go, build, -o, "${artifact}", ../tools/wordcount]

defaults:
  warmup: 1
  runs: 10

benchmarks:
  - name: small
    tags: [smoke]
    setup:
      - command: ["${artifact}", -gen, "100", -o, "${workdir}/input.txt"]
    stdin: "${workdir}/input.txt"
    baseline: scanner
    commands:
      scanner:
        command: ["${artifact}", -impl, scanner]
      readall:
        command: ["${artifact}", -impl, readall]

  - name: medium
    setup:
      - command: ["${artifact}", -gen, "10000", -o, "${workdir}/input.txt"]
    stdin: "${workdir}/input.txt"
    baseline: scanner
    commands:
      scanner:
        command: ["${artifact}", -impl, scanner]
      readall:
        command: ["${artifact}", -impl, readall]

  - name: large
    setup:
      - command: ["${artifact}", -gen, "200000", -o, "${workdir}/input.txt"]
    stdin: "${workdir}/input.txt"
    baseline: scanner
    commands:
      scanner:
        command: ["${artifact}", -impl, scanner]
      readall:
        command: ["${artifact}", -impl, readall]

$ himorime run examples/input-sizes --tag smoke

Without --tag there are three benchmarks, small, medium and large, each with both implementations, followed by the geometric mean of each command’s ratio to the baseline across the three cases.

  • setup runs once per benchmark, before warmup, and writes the fixture into that benchmark’s own ${workdir}.
  • Read the per-case rows first. The geometric mean is a summary and hides that an implementation can win on small inputs and lose on large ones.

Example: examples/input-sizes