About

Why sqly exists #

sqly was built to make large CSV files easy to check.

In a project between 2022 and 2025, an app’s master data lived in CSV files:

• large — over 20,000 rows by 300 columns, or 100,000 rows
• read by a Go program that inserted the records into several DB tables
• not one-to-one with those tables: one CSV fed several
• edited by several people, none of them engineers
• updated several times a month

Two things made that painful. Excel, Numbers, and Google Sheets take a long time to open a file that size, and often crash on it. And when a value has the wrong type — a string where a number belongs — the import fails with “a decode error occurred”, without saying which column. Finding the bad column among 300 by hand, in a spreadsheet, is not an engineer’s job.

So: query the file with SQL instead.

The name #

sqly was named to surpass the famous jmoiron/sqlx — x, then y. That is a joke. The real origin is the slangy sense of “SQL on CSV? seriously?”.

How it is built #

sqly reads each file, converts it to a table, and stores it in an in-memory SQLite3 database. It has no SQL parser of its own; parsing and execution are SQLite’s, which is why the full query engine — CTEs, window functions, joins, aggregates — is available on a CSV file.

Two libraries carry most of the work, both from the same author:

• filesql — a database/sql driver that loads CSV, TSV, LTSV, JSON, JSONL, Parquet, Excel, ACH, and Fedwire files into SQLite, and writes them back. It also holds the dialect translation behind --dialect.
• prompt — the line editor behind the interactive shell: completion, history, multi-line input, and raw-mode handling across Unix and Windows.

The project’s layering is checked in CI with go-arch-lint, against the rules in .go-arch-lint.yml.

Contributing #

Issues and pull requests are welcome; see CONTRIBUTING.md and how to build and test. A GitHub Star also motivates development.

Benchmark #

Historical measurement from sqly v0.30.0 on the environment shown below. It is not a performance guarantee for the current release. The figures have not been re-measured since; treat them as a rough sense of scale rather than as a number to plan against, and run make bench on your own machine if the answer matters.

make bench measures one full run (import the CSV into the in-memory DB, then run the query) over testdata/benchmark/customers100000.csv (100,000 rows, 12 columns):

Measured on an AMD Ryzen 7 5800U, Go 1.25, sqly v0.30.0. The comparison below comes from the same run, so both are refreshed together rather than at each release.

The same query on the same file (top 10 countries by row count), best of 5 end-to-end runs:

sqly stays in the same sub-second range as the CSV-focused tools while reading the widest set of formats, shipping an interactive shell, and building as a pure-Go binary with no CGO.