Real CLIs tested with atago
These real programs run their end-to-end suites on atago; most were migrated from ShellSpec. The generated behavior docs live in doc/e2e/.
Every suite here was written by the atago project on its own initiative and runs in atago’s own CI — the point is to exercise atago against real programs of every shape. The third-party projects listed below are not affiliated with atago, did not commission these tests, and these specs are not their official test suites.
Programs maintained by the author, starting with atago itself (test/e2e/tools/):
| Tool | Feature | Specs | Docs |
|---|---|---|---|
| atago | atago tested by atago: the self-hosted specs run the real built binary in CI (make e2e). | specs | docs |
| gup | Updates and manages the Go command-line tools in $GOBIN. | specs | docs |
| sqly | Runs SQL against CSV/TSV/LTSV/JSON/Parquet/Excel/ACH/Fedwire files. | specs | docs |
| truss | Image transformation (convert/resize/re-encode). | specs | docs |
| iso8583tool | Debugs and inspects ISO 8583 payment messages. | specs | docs |
| jose | Signs and encrypts with JOSE. | specs | docs |
| career | Renders résumé PDFs from a single YAML file. | specs | docs |
| mimixbox | Packs many Unix commands into one BusyBox-style binary. | specs | docs |
| mobilepkg | Inspects Android packages for metadata and security findings. | specs | docs |
Third-party programs (test/e2e/thirdparty/). atago runs each as an unmodified binary and ships only its own spec YAML — no third-party code is copied or redistributed, which every listed license permits:
| Program | Feature | License | Specs | Docs |
|---|---|---|---|---|
| git | Version control — runs in CI on all three OSes: init/add/commit and status, plus diff --exit-code as a status contract, exact --porcelain codes, a fast-forward merge and a real conflict, stash and patch round trips proven byte for byte, .gitignore keeping a file out of the index, a local clone reproducing the same commit, and the documented failures (no repository, duplicate tag, nothing staged). | GPL-2.0 | specs | docs |
| jq | JSON processor: stdin-driven filters, –arg injection, and its documented exit-code contract (1 for -e false, 3 for a bad program, parse failures). | MIT | specs | docs |
| fzf | Interactive fuzzy finder driven inside a real pseudo-terminal: expect/send sessions type queries, multi-select, and abort (exit 130); –filter pins the non-TTY contract. | MIT | specs | docs |
| fx | JSON viewer that is a filter when given a reducer and a full-screen tree when not. The filter half pins the JavaScript semantics a jq user gets wrong (a missing key is undefined on stderr with exit 0, dereferencing through it is exit 1), exit() choosing the process’s code, and the identity path keeping number literals a float64 round trip would rewrite; parser flags cover YAML/TOML/raw/slurp, filename-driven format detection, --strict rejecting comments and trailing commas, and base64 and YAML round trips. save is pinned with exhaustive changes: deltas — in-place rewrite, idempotence, the no-file and symlink refusals — alongside .fxrc.js loaded from the workdir and from a sandboxed home that stays unwritten. The viewer runs in a pty: folding, regexp search reporting the cursor’s JSON path, and the split that makes fx data.json > picked.txt work, with the interface drawn on stderr and only the chosen value on stdout. | MIT | specs | docs |
| gum | Shell-script UI toolkit: non-interactive stdout contracts (version-check, join, table, log) plus TTY prompts and pickers (choose, filter, confirm, input, file, pager) driven over a real terminal. | MIT | specs | docs |
| Helix Editor | Modal editor workflows in a real TUI: append/insert editing, line-open-above/below, undo-after-delete, nested-path saves, multi-buffer next/previous/edit-both flows, copy-via-:w path, and UTF-8 filename/content saves under isolated Helix config/cache homes. | MPL-2.0 | specs | docs |
| lazygit | Git TUI on a real repository: staged/unstaged/untracked transitions (including second-item selection), editor-driven commits (including pre-staged changes), discard accept/cancel flows, branch creation including slash names and canceled prompts, stash create/pop plus cancel paths, and the invalid nested-path startup contract. | MIT | specs | docs |
| htop | Full-screen alternate-screen TUI as a pty screen-rendering testbed: the rendered frame shows the meters and column header, q quits with a clean lifecycle, and an unknown flag is rejected before the TUI opens. | GPL-2.0 | specs | docs |
| yazi | Terminal file manager TUI: rendered-screen hidden-file toggles and live filtering; file/dir create + rename including space-bearing names; explicit ENTRIES... startup for files, directories, hidden paths, and multi-tab launches; chooser-file and cwd-file handoff across tab switches, close flows, and explicit selection/cancel-selection paths; absolute/relative symlinks plus hardlinks via the physical Ctrl+- key; single-file and directory copy/move flows into sibling directories, including overwrite/collision suffixing and spaced-path variants; visual/space/ctrl-a/ctrl-r selection semantics across file and directory operations; trash vs permanent delete into an isolated sandbox home, including canceled dialogs; modified-time plus alphabetical/extension/size/natural sort checks on both the rendered list and chooser output; content search via S; file-info toggles via Tab; arrow-key navigation and tab reordering/switching; cancel-yank flows via X/Y; and blocking : shell-command integration for file, dir, symlink, and hidden-file mutations. | MIT | specs | docs |
| redis | Server+client pair: redis-server as a scenario service (readiness by log and by port), redis-cli round-trips (PING/SET/GET/INCR/TTL), error contracts, and a polled graceful shutdown. | RSALv2/SSPLv1 | specs | docs |
| hugo | Scaffold + build + serve in one binary: hugo new site tree assertions, a theme-less --minify build, and hugo server bootstrapped in suite.setup then queried over HTTP. | Apache-2.0 | specs | docs |
| openssl | Cryptography toolkit: exact digests, keygen → sign → verify (and tamper detection), encrypt/decrypt round-trips with the wrong-password failure mode, self-signed certificates. | Apache-2.0 | specs | docs |
| sqlite3 | Embedded-database shell driven as a real binary: one-shot SQL, -json/-csv output modes, .dump → .read rebuilds, .import from CSV, bad-SQL diagnostics. | Public Domain | specs | docs |
| caddy | Self-hosted web server, booted from an authored Caddyfile and queried over HTTP. | Apache-2.0 | specs | docs |
| coredns | Self-hosted DNS server: an authored zone queried with real dig — authoritative answers, CNAME chasing, NXDOMAIN/REFUSED, and the health plugin over HTTP. | Apache-2.0 | specs | docs |
| gitea | Self-hosted git service: booted with SQLite, administered via its CLI, driven over the REST API (repos, commits, issues), then cloned with real git. | MIT | specs | docs |
| gotify | Self-hosted notification server: app provisioning, token-authenticated pushes, and the app icon uploaded as real multipart/form-data, downloaded back, and verified as a PNG. | MIT | specs | docs |
| grafana | Self-hosted observability platform: health/build info, the login redirect asserted with follow_redirects: false, and a dashboard + datasource lifecycle over the REST API. | AGPL-3.0 | specs | docs |
| mailpit | Self-hosted email testing: messages delivered over real SMTP (stock curl), then asserted via the REST API — capture, search, MIME attachments, teardown. | MIT | specs | docs |
| minio | Self-hosted S3-compatible object storage: full object lifecycle via mc, versioning, anonymous bucket policies, S3 XML error contract. | AGPL-3.0 | specs | docs |
| aws-cli | The AWS CLI driven against a local MinIO S3 endpoint — a cloud CLI tested offline: bucket/object lifecycle, byte-identical round-trip, head-object JSONPath, presigned-URL fetch, missing-key error contract. | Apache-2.0 | specs | docs |
| python3 | The REPL as an interactive pty testbed: prompt detection, multi-exchange expect/send, EOF exit, traceback recovery, and TTY branching — a copy-paste template for testing your own REPL. | PSF-2.0 | specs | docs |
| ssh-keygen | OpenSSH key generation: non-interactive and interactive-passphrase (pty) key pairs, generated-file asserts, the exact fingerprint contract, and verification-failure exit codes. | BSD | specs | docs |
| ffmpeg | Media pipeline: lavfi video synthesis, ffprobe stream JSON (JSONPath), frame extraction verified with image assertions (format/dimensions/pixel similarity), a webm transcode, and error contracts. | LGPL/GPL | specs | docs |
| pandoc | Document conversion: markdown→HTML, a docx artifact verified by unzipping its OOXML, a stdin→stdout filter (via stdin: {file:}), the JSON AST queried with JSONPath, metadata-driven standalone output, a markdown↔HTML round-trip, and the unknown-format and missing-input error contracts. | GPL-2.0+ | specs | docs |
| ImageMagick | Image conversion and inspection: every output decoded and checked as an image (format sniffed from bytes, pixel dimensions, alpha), aspect-ratio versus forced resize, a PNG↔PPM round trip verified both by atago pixel comparison and by magick compare -metric AE as an independent oracle, metadata stripping that leaves the picture untouched, the identify JSON contract, and the missing/corrupt-input failure contracts. | ImageMagick (Apache-2.0 style) | specs | docs |
| zstd | Compression round trips proven byte for byte with equals_file (including an empty file and NUL bytes), the input kept unless --rm, an existing archive never silently overwritten and -f overriding that, -t integrity verification of a good and a corrupt frame, a failed decompression leaving no partial output, the stdin→stdout pipeline, and the missing-input and unknown-option failure contracts. | BSD-3-Clause / GPL-2.0 | specs | docs |
| zip / unzip | Extracted trees asserted with dir: in every mode (recursive membership, a file-only count, a glob, a snapshot manifest), round trips proven byte for byte with equals_file including an empty file and NUL bytes, -j flattening as the metamorphic pair of the tree case, the overwrite decision (declined by default, -n, -o), archive update and entry deletion, the documented exit codes 0/1/9/10/11 and zip’s 12, and the zip-slip refusal: a ../ or absolute entry is stripped, reported, and kept inside the destination, proven by changes: rather than by the warning text. | Info-ZIP | specs | docs |
| Ghostscript | PostScript→PDF→text/raster pipeline asserted on the documents themselves: page count, the text really inside them, and Info-dictionary metadata via pdf:, plus the rasterized page via image:. Relationships are pinned too — one selected page keeps its own text and drops the other, merging sums the pages, txtwrite agrees with the extracted text, and a repeat conversion yields identical text — alongside the missing-input and unknown-device failures (whose explanations go to stdout) and the surprise that a failed conversion still leaves a PDF behind. | AGPL-3.0 | specs | docs |
| terraform | The multi-exit-code IaC CLI, fully offline via the builtin terraform_data resource: init/validate/plan/apply/destroy, the plan -detailed-exitcode 0/1/2 contract, state JSON, and fmt -check exit 3. | BUSL-1.1 | specs | docs |
| age | Modern file encryption: keygen, binary-safe encrypt/decrypt round-trips, armored (PEM) output, interactive-passphrase mode (pty), and clean failure semantics. | BSD-3-Clause | specs | docs |
| sops | Secrets encryption with age as the key backend: value-only encryption that keeps keys readable and adds a MAC, an encrypt→decrypt value round-trip, single-field --extract, wrong-key failure (128), a tampered ciphertext failing the MAC (25), and --encrypted-regex scoping which keys are encrypted. Every age key is generated in the workdir. | MPL-2.0 | specs | docs |
| kustomize | Declarative Kubernetes manifest rendering, fully offline and deterministic: build transforms (prefix/namespace/labels/images), generator content-hash suffixes (and disabling them), strategic-merge and JSON6902 patches, the load-restrictor security contract, byte-identical repeated builds, and the create/edit/fix authoring side effects. | Apache-2.0 | specs | docs |
| actionlint | Static linter for GitHub Actions workflows, offline and deterministic: a clean workflow exits 0 silently, distinct rule categories (undefined needs, unknown runner label, invalid expression) surface as tagged findings on stdout, the JSON format is a structured oracle over them, -ignore suppresses a match, and stdin mode lints piped content. | MIT | specs | docs |
| aqua | Declarative CLI version manager: install pulls a tool from a local registry + local HTTP file server, lands it executable under the managed root, and aqua exec runs it — its essential install→resolve→run flow, hermetic. Plus the offline surface: init (idempotent), policy init, root-dir/completion generators, and exit codes separating a usage error (3) from a runtime error (1, which on an unmanaged command). | MIT | specs | docs |
| ecspresso | Amazon ECS deploy tool: a real deploy against a moto mock AWS endpoint registers a task definition and creates the service, then a changed image rolls it to a new revision — the ECS state change observed with aws ecs describe-services. Plus the offline render surface: env/must_env template functions and defaults, a jsonnet task definition with --ext-str, config resolution, and failure modes at distinct exit codes, checked as structured JSON. | MIT | specs | docs |
| nats | Self-hosted messaging: request/reply through the real broker, JetStream persistence (create → publish → count → purge), the KV store, and the monitoring endpoint. | Apache-2.0 | specs | docs |
| ntfy | Self-hosted push notifications: publish with headers, poll the JSON feed, topic isolation, and deny-all access control unlocked via the admin CLI. | Apache-2.0 | specs | docs |
| prometheus | Self-hosted monitoring: promtool config/rule checks and rule unit tests, the query API, and a self-scrape polled with http retry. | Apache-2.0 | specs | docs |
| pushgateway | Self-hosted metrics gateway: raw text body: pushes scraped back with their group labels, a malformed payload rejected (400) and never ingested, POST-merges-vs-PUT-replaces group semantics, an added instance label, and job-group deletion. | Apache-2.0 | specs | docs |
| rclone | Self-hosted file sync: copy/sync/check semantics (including the corruption failure mode), JSON listings, and rclone serve http. | MIT | specs | docs |
| restic | Self-hosted backup: init → backup → restore round-trip, snapshot JSON, diff, integrity check, retention, wrong-password failure mode. | BSD-2-Clause | specs | docs |
| transfer.sh | Self-hosted file sharing: a PNG uploaded as the raw request body (body_file), downloaded back to disk (body_to), and verified byte-for-byte; multipart uploads too. | MIT | specs | docs |
| webhook | Self-hosted webhook receiver driven by config fixtures: a request runs a command whose file side effect and output are both checked, a value trigger-rule gates execution (and blocks it, proven by the absent side effect), an HMAC signature verified against an openssl-computed oracle, the http-methods allowlist (405), and a non-zero command surfaced as 500. | MIT | specs | docs |