Skip to content
Back to search

sq-io

MarketplaceWorks with Claude CodeGitHub

sq data wrangler

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/neilotoole/sq with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/neilotoole/sq'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/neilotoole--sq/badge.svg)](/listings/neilotoole--sq)

HTML

<a href="/listings/neilotoole--sq"><img src="/api/listings/neilotoole--sq/badge.svg" alt="findsafeskills safety rating" /></a>

Install

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

macOS

brew install sq

> [!IMPORTANT] > `sq` is now a [core brew formula](https://formulae.brew.sh/formula/sq#default). Previously, > `sq` was available via `brew install neilotoole/sq/sq`. If you have installed `sq` this way, > you should uninstall it (`brew uninstall neilotoole/sq/sq`) before installing the new formula > via `brew install sq`.

Linux

/bin/sh -c "$(curl -fsSL https://sq.io/install.sh)"

Windows

scoop bucket add sq https://github.com/neilotoole/sq
scoop install sq

Source

Prefer a package manager above, or a [release binary](https://github.com/neilotoole/sq/releases). `go install` needs the SQLite build tags, and its binary reports version `v0.0.0-dev`; use `make install` from a clone for a version-stamped build.

go install -tags "sqlite_vtable sqlite_stat4 sqlite_fts5 sqlite_introspect sqlite_json sqlite_math_functions" \
  github.com/neilotoole/sq@latest

Docker

The [`ghcr.io/neilotoole/sq`](https://github.com/neilotoole/sq/pkgs/container/sq) image is preloaded with `sq` and a handful of related tools like `jq`. The same image is published to Docker Hub as [`neilotoole/sq`](https://hub.docker.com/r/neilotoole/sq). Both are cosign-signed and carry SLSA build provenance and an SPDX SBOM.

Local

# Shell into a one-time container.
$ docker run -it ghcr.io/neilotoole/sq zsh

# Start detached (background) container named "sq-shell".
$ docker run -d --name sq-shell ghcr.io/neilotoole/sq
# Shell into that container.
$ docker exec -it sq-shell zsh

Kubernetes

Running `sq` in a Kubernetes environment is useful for DB migrations, as well as general data wrangling.

# Start pod named "sq-shell".
$ kubectl run sq-shell --image ghcr.io/neilotoole/sq
# Shell into the pod.
$ kubectl exec -it sq-shell -- zsh

See other [install options](https://sq.io/docs/install/).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 26 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. {
  2. "name": "sq-io",
  3. "owner": {
  4. "name": "Neil O'Toole",
  5. "email": "neilotoole@apache.org"
  6. },
  7. "metadata": {
  8. "description": "Claude Code marketplace for sq (data wrangler) — CLI skills and references."
  9. },
  10. "plugins": [
  11. {
  12. "name": "sq",
  13. "source": "./",
  14. "description": "Agent skill for using the sq CLI after install; progressive driver references and links to sq.io docs.",
  15. "version": "1.0.0",
  16. "author": {
  17. "name": "Todd Papaioannou",
  18. "email": "drluckyspin@gmail.com"
  19. },
  20. "homepage": "https://sq.io",
  21. "repository": "https://github.com/neilotoole/sq",
  22. "license": "MIT",
  23. "keywords": ["sq", "sql", "data", "cli", "slq"]
  24. }
  25. ]
  26. }
README.md · README · 441 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # sq data wrangler
  2. [![Go Reference](https://pkg.go.dev/badge/github.com/neilotoole/sq.svg)](https://pkg.go.dev/github.com/neilotoole/sq)
  3. [![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://github.com/neilotoole/sq/blob/master/LICENSE)
  4. ![Main pipeline](https://github.com/neilotoole/sq/actions/workflows/main.yml/badge.svg)
  5. [![codecov](https://codecov.io/gh/neilotoole/sq/graph/badge.svg)](https://codecov.io/gh/neilotoole/sq)
  6. `sq` is a command line tool that provides [jq](https://jqlang.org)-style access to
  7. structured data sources: SQL databases, or document formats like CSV or Excel.
  8. It is the lovechild of SQL+jq.
  9. ![sq](site/static/images/repo/splash.png)
  10. `sq` executes jq-like [queries](https://sq.io/docs/query), or database-native
  11. [SQL](https://sq.io/docs/cmd/sql/). It can [join](https://sq.io/docs/query#cross-source-joins)
  12. across sources: join a CSV file to a Postgres table, or MySQL with Excel.
  13. `sq` outputs to a multitude of [formats](https://sq.io/docs/output#formats)
  14. including [JSON](https://sq.io/docs/output#json),
  15. [Excel](https://sq.io/docs/output#xlsx), [CSV](https://sq.io/docs/output#csv),
  16. [HTML](https://sq.io/docs/output#html), [Markdown](https://sq.io/docs/output#markdown)
  17. and [XML](https://sq.io/docs/output#xml), and can [insert](https://sq.io/docs/output#insert) query
  18. results directly to a SQL database.
  19. `sq` can also [inspect](https://sq.io/docs/inspect) sources to view metadata about the
  20. source structure (tables, columns, size). You can use [`sq diff`](https://sq.io/docs/diff)
  21. to compare tables, or entire databases. `sq` has commands for common database operations to
  22. [copy](https://sq.io/docs/cmd/tbl-copy), [truncate](https://sq.io/docs/cmd/tbl-truncate),
  23. and [drop](https://sq.io/docs/cmd/tbl-drop) tables.
  24. Find out more at [sq.io](https://sq.io).
  25. > [!TIP]
  26. > The rest of this doc is mainly for `sq` end users and first-timers. Contributors
  27. > (bug reports, feature requests, pull requests), see [CONTRIBUTING.md](./CONTRIBUTING.md).
  28. ## Drivers
  29. `sq` knows how to deal with a data source type via a [driver](https://sq.io/docs/drivers)
  30. implementation. To view the installed/supported drivers:
  31. ```shell
  32. $ sq driver ls
  33. DRIVER DESCRIPTION
  34. sqlite3 SQLite
  35. rqlite rqlite
  36. duckdb DuckDB
  37. postgres PostgreSQL
  38. sqlserver Microsoft SQL Server
  39. mysql MySQL
  40. clickhouse ClickHouse
  41. oracle Oracle
  42. csv Comma-Separated Values
  43. tsv Tab-Separated Values
  44. json JSON
  45. jsona JSON Array: LF-delimited JSON arrays
  46. jsonl JSON Lines: LF-delimited JSON objects
  47. xlsx Microsoft Excel XLSX
  48. ```
  49. ## Install
  50. ### macOS
  51. ```shell
  52. brew install sq
  53. ```
  54. > [!NOTE]
  55. > `sq` is a [core brew formula](https://formulae.brew.sh/formula/sq#default). The old
  56. > `neilotoole/sq` tap is retired. If you installed `sq` from it (`brew install neilotoole/sq/sq`),
  57. > the next `brew update` switches that install to the core formula.
  58. >
  59. > Homebrew no longer builds bottles for Intel Macs, so on an Intel Mac, `brew install sq` builds
  60. > from source. To skip the build, download the `macos-amd64` binary from
  61. > [GitHub releases](https://github.com/neilotoole/sq/releases).
  62. ### Linux
  63. ```shell
  64. /bin/sh -c "$(curl -fsSL https://sq.io/install.sh)"
  65. ```
  66. ### Windows
  67. ```shell
  68. scoop bucket add sq https://github.com/neilotoole/sq
  69. scoop install sq
  70. ```
  71. ### Source
  72. Prefer a package manager above, or a [release binary](https://github.com/neilotoole/sq/releases).
  73. `go install` needs the SQLite build tags, and its binary reports version `v0.0.0-dev`; use
  74. `make install` from a clone for a version-stamped build.
  75. ```shell
  76. go install -tags "sqlite_vtable sqlite_stat4 sqlite_fts5 sqlite_introspect sqlite_json sqlite_math_functions" \
  77. github.com/neilotoole/sq@latest
  78. ```
  79. ### Docker
  80. The [`ghcr.io/neilotoole/sq`](https://github.com/neilotoole/sq/pkgs/container/sq)
  81. image is preloaded with `sq` and a handful of related tools like `jq`. The same
  82. image is published to Docker Hub as
  83. [`neilotoole/sq`](https://hub.docker.com/r/neilotoole/sq). Both are cosign-signed
  84. and carry SLSA build provenance and an SPDX SBOM.
  85. #### Local
  86. ```shell
  87. # Shell into a one-time container.
  88. $ docker run -it ghcr.io/neilotoole/sq zsh
  89. # Start detached (background) container named "sq-shell".
  90. $ docker run -d --name sq-shell ghcr.io/neilotoole/sq
  91. # Shell into that container.
  92. $ docker exec -it sq-shell zsh
  93. ```
  94. #### Kubernetes
  95. Running `sq` in a Kubernetes environment is useful for DB migrations,
  96. as well as general data wrangling.
  97. ```shell
  98. # Start pod named "sq-shell".
  99. $ kubectl run sq-shell --image ghcr.io/neilotoole/sq
  100. # Shell into the pod.
  101. $ kubectl exec -it sq-shell -- zsh
  102. ```
  103. See other [install options](https://sq.io/docs/install/).
  104. ## Agent skills
  105. An [Agent Skills](https://agentskills.io/) skill for this repo lives under
  106. [`skills/sq/`](./skills/sq/) and helps coding assistants use an installed `sq`
  107. (CLI patterns, `references/` for drivers). **User-facing install and usage** are
  108. documented at **[sq.io/docs/agent-skills](https://sq.io/docs/agent-skills)**.
  109. Quick install from GitHub:
  110. ```shell
  111. npx skills add neilotoole/sq
  112. ```
  113. ## Documentation site ([sq.io](https://sq.io))
  114. User-facing docs at [sq.io](https://sq.io) are built from the [`site/`](./site/) directory
  115. in this repository (Hugo + Bun + Netlify). To work on the website locally, see
  116. [`site/README.md`](./site/README.md). The standalone **`sq-web`** repository is **archived**;
  117. open issues and pull requests here against **`neilotoole/sq`** for doc and site changes.
  118. ## Overview
  119. Use `sq help` to see command help. Docs are over at [sq.io](https://sq.io).
  120. Read the [overview](https://sq.io/docs/overview/), and
  121. [tutorial](https://sq.io/docs/tutorial/). The [cookbook](https://sq.io/docs/cookbook/) has
  122. recipes for common tasks, and the [query guide](https://sq.io/docs/query) covers `sq`'s query language.
  123. The major concept is: `sq` operates on data sources, which are treated as SQL databases (even if the
  124. source is really a CSV or XLSX file etc.).
  125. In a nutshell, you [`sq add`](https://sq.io/docs/cmd/add) a source (giving it a
  126. [`handle`](https://sq.io/docs/concepts#handle)), and then execute commands against the source.
  127. ### Sources
  128. Initially there are no [sources](https://sq.io/docs/source).
  129. ```shell
  130. $ sq ls
  131. ```
  132. Let's [add](https://sq.io/docs/cmd/add) a source. First we'll add a
  133. [SQLite](https://sq.io/docs/drivers/sqlite) database, but this could also be
  134. [Postgres](https://sq.io/docs/drivers/postgres),
  135. [SQL Server](https://sq.io/docs/drivers/sqlserver),
  136. [MySQL](https://sq.io/docs/drivers/mysql), [Oracle](https://sq.io/docs/drivers/oracle) etc.,
  137. or a document source such as [Excel](https://sq.io/docs/drivers/xlsx) or
  138. [CSV](https://sq.io/docs/drivers/csv).
  139. Download the sample DB, and `sq add` the source.
  140. ```shell
  141. $ wget https://sq.io/testdata/sakila.db
  142. $ sq add ./sakila.db
  143. @sakila sqlite3 sakila.db
  144. $ sq ls -v
  145. HANDLE ACTIVE DRIVER LOCATION OPTIONS
  146. @sakila active sqlite3 sqlite3:///Users/demo/sakila.db
  147. $ sq ping @sakila
  148. @sakila 1ms pong
  149. $ sq src
  150. @sakila sqlite3 sakila.db
  151. ```
  152. The [`sq ping`](https://sq.io/docs/cmd/ping) command simply pings the source
  153. to verify that it's available.
  154. [`sq src`](https://sq.io/docs/cmd/src) lists the
  155. [_active source_](https://sq.io/docs/source#active-source), which in our case is `@sakila`.
  156. You can change the active source using `sq src @other_src`.
  157. When there's an active source specified, you can usually omit the handle from `sq` commands.
  158. Thus you could instead do:
  159. ```shell
  160. $ sq ping
  161. @sakila 1ms pong
  162. ```
  163. > [!TIP]
  164. > Document sources such as CSV or Excel can be added from the local filesystem, or
  165. > from an HTTP URL.
  166. >
  167. > ```shell
  168. > $ sq add https://acme.s3.amazonaws.com/sales.csv
  169. > ```
  170. >
  171. > ![sq inspect remote](site/static/images/repo/sq_inspect_remote_s3.png)
  172. > See the [sources](https://sq.io/docs/source#download) docs for more.
  173. ### Query
  174. Fundamentally, `sq` is for querying data. The jq-style syntax is covered in
  175. detail in the [query guide](https://sq.io/docs/query).
  176. ![sq query where slq](site/static/images/repo/sq_query_where_slq.png)
  177. The above query selected some rows from the `actor` table. You could also
  178. use [native SQL](https://sq.io/docs/cmd/sql), e.g.:
  179. ![sq query where sql](site/static/images/repo/sq_query_where_sql.png)
  180. But we're flying a bit blind here: how did we know about the `actor` table?
  181. ### Inspect
  182. [`sq inspect`](https://sq.io/docs/inspect) is your friend.
  183. ![sq inspect](site/static/images/repo/sq_inspect_source_text.png)
  184. Use [`sq inspect -v`](https://sq.io/docs/cmd/inspect) to see more detail.
  185. Or use [`-j`](https://sq.io/docs/output#json) to get JSON output:
  186. ![sq inspect -j](site/static/images/repo/sq_inspect_sakila_sqlite_json.png)
  187. Combine `sq inspect` with [jq](https://jqlang.github.io/jq/) for some useful capabilities.
  188. Here's how to [list](https://sq.io/docs/cookbook#list-table-names)
  189. all the table names in the active source:
  190. ```shell
  191. $ sq inspect -j | jq -r '.tables[] | .name'
  192. actor
  193. address
  194. category
  195. city
  196. country
  197. customer
  198. [...]
  199. ```
  200. And here's how you
  201. could [export](https://sq.io/docs/cookbook#export-all-table-data-to-csv) each table
  202. to a CSV file:
  203. ```shell
  204. $ sq inspect -j | jq -r '.tables[] | .name' | xargs -I % sq .% --csv --output %.csv
  205. $ ls
  206. actor.csv city.csv customer_list.csv film_category.csv inventory.csv rental.csv staff.csv
  207. address.csv country.csv film.csv film_list.csv language.csv sales_by_film_category.csv staff_list.csv
  208. category.csv customer.csv film_actor.csv film_text.csv payment.csv sales_by_store.csv store.csv
  209. ```
  210. Note that you can also inspect an individual table:
  211. ![sq inspect actor verbose](site/static/images/repo/sq_inspect_actor_verbose.png)
  212. Read more about [`sq inspect`](https://sq.io/docs/inspect).
  213. ### Diff
  214. Use [`sq diff`](https://sq.io/docs/diff) to compare metadata, or row data, for sources,
  215. or individual tables.
  216. The default behavior is to diff table schema and row counts. Table row data is not compared
  217. in this mode.
  218. ![sq diff](site/static/images/repo/sq_diff_src_default.png)
  219. Use [`--data`](https://sq.io/docs/diff#--data) to compare row data.
  220. ![sq diff data](site/static/images/repo/sq_diff_table_data.png)
  221. There are many more options available. See the [diff docs](https://sq.io/docs/diff).
  222. ### Insert query results
  223. `sq` query results can be [output](https://sq.io/docs/output) in various formats
  224. ([`text`](https://sq.io/docs/output#text),
  225. [`json`](https://sq.io/docs/output#json),
  226. [`csv`](https://sq.io/docs/output#csv), etc.). Those results can also be "outputted"
  227. as an [_insert_](https://sq.io/docs/output#insert) into a database table.
  228. That is, you can use `sq` to insert results from a Postgres query into a MySQL table,
  229. or copy an Excel worksheet into a SQLite table, or a push a CSV file into
  230. a SQL Server table etc.
  231. > [!TIP]
  232. > If you want to copy a table inside the same (database) source,
  233. > use [`sq tbl copy`](https://sq.io/docs/cmd/tbl-copy) instead, which uses the database's
  234. > native table copy functionality.
  235. Here we query a CSV file, and insert the results into a Postgres table.
  236. ![sq query insert inspect](site/static/images/repo/sq_query_insert_inspect.png)
  237. ### Cross-source joins
  238. `sq` can perform the usual [joins](https://sq.io/docs/query#joins). Here's how you would
  239. join tables `actor`, `film_actor`, and `film`:
  240. ```shell
  241. $ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id) | .first_name, .last_name, .title'
  242. ```
  243. But `sq` can also join across data sources. That is, you can join an Excel worksheet with a
  244. Postgres table, or join a CSV file with MySQL, and so on.
  245. This example joins a Postgres database, an Excel worksheet, and a CSV file.
  246. ![sq join multi source](site/static/images/repo/sq_join_multi_source.png)
  247. Read more about cross-source joins in the [query guide](https://sq.io/docs/query#joins).
  248. ### Table commands
  249. `sq` provides several handy commands for working with tables:
  250. [`tbl copy`](https://sq.io/docs/cmd/tbl-copy), [`tbl truncate`](https://sq.io/docs/cmd/tbl-truncate)
  251. and [`tbl drop`](https://sq.io/docs/cmd/tbl-drop).
  252. Note that these commands work directly
  253. against SQL database sources, using their native SQL commands.
  254. ```shell
  255. $ sq tbl copy .actor .actor_copy
  256. Copied table: @sakila.actor --> @sakila.actor_copy (200 rows copied)
  257. $ sq tbl truncate .actor_copy
  258. Truncated 200 rows from @sakila.actor_copy
  259. $ sq tbl drop .actor_copy
  260. Dropped table @sakila.actor_copy
  261. ```
  262. ### UNIX pipes
  263. For file-based sources (such as CSV or XLSX), you can `sq add` the source file,
  264. but you can also pipe it:
  265. ```shell
  266. $ cat ./example.xlsx | sq .Sheet1
  267. ```
  268. Similarly, you can inspect:
  269. ```shell
  270. $ cat ./example.xlsx | sq inspect
  271. ```
  272. ## Output formats
  273. `sq` has many [output formats](https://sq.io/docs/output):
  274. - `--text`: [Text](https://sq.io/docs/output#text)
  275. - `--json`: [JSON](https://sq.io/docs/output#json)
  276. - `--jsona`: [JSON Array](https://sq.io/docs/output#jsona)
  277. - `--jsonl`: [JSON Lines](https://sq.io/docs/output#jsonl)
  278. - `--csv` / `--tsv` : [CSV](https://sq.io/docs/output#csv) / [TSV](https://sq.io/docs/output#tsv)
  279. - `--xlsx`: [XLSX](https://sq.io/docs/output#xlsx) (Microsoft Excel)
  280. - `--html`: [HTML](https://sq.io/docs/output#html)
  281. - `--xml`: [XML](https://sq.io/docs/output#xml)
  282. - `--yaml`: [YAML](https://sq.io/docs/output#yaml)
  283. - `--markdown`: [Markdown](https://sq.io/docs/output#markdown)
  284. - `--raw`: [Raw](https://sq.io/docs/output#raw) (bytes)
  285. ## Contributing
  286. Bug reports, feature requests, and pull requests are welcome. Start with
  287. [CONTRIBUTING.md](./CONTRIBUTING.md). Contributor and maintainer docs
  288. (architecture, drivers, grammar, workflows) are indexed in
  289. [`docs/README.md`](./docs/README.md), and [AGENTS.md](./AGENTS.md) covers
  290. AI-agent conventions.
  291. ## CHANGELOG
  292. See [CHANGELOG.md](./CHANGELOG.md).
  293. ## Acknowledgements
  294. - Thanks to [Diego Souza](https://github.com/diegosouza) for creating
  295. the [Arch Linux package](https://aur.archlinux.org/packages/sq-bin), and [`@icp`](https://github.com/icp1994)
  296. for creating the [Void Linux package](https://github.com/void-linux/void-packages/blob/master/srcpkgs/sq/template).
  297. - Much inspiration is owed to [jq](https://jqlang.github.io/jq/).
  298. - See [`go.mod`](https://github.com/neilotoole/sq/blob/master/go.mod) for a list of third-party
  299. packages.
  300. - `sq` imports a bunch of [`usql`](https://github.com/xo/usql) functionality.
  301. - Additionally, `sq` incorporates a modified version of
  302. [`olekukonko/tablewriter`](https://github.com/olekukonko/tablewriter).
  303. - The [_Sakila_](https://dev.mysql.com/doc/sakila/en/) example databases were lifted
  304. from [jOOQ](https://github.com/jooq/jooq), which in turn owe their heritage to earlier work on
  305. Sakila.
  306. - Date rendering via [`ncruces/go-strftime`](https://github.com/ncruces/go-strftime).
  307. - A modified version [`dolmen-go/contextio`](https://github.com/dolmen-go/contextio) is
  308. incorporated into the codebase.
  309. - [`djherbis/buffer`](https://github.com/djherbis/buffer) is used for caching.
  310. - A forked version of [`nightlyone/lockfile`](https://github.com/nightlyone/lockfile) is incorporated.
  311. - The human-friendly `text` log format handler is a fork of [`lmittmann/tint`](https://github.com/lmittmann/tint).
  312. ## Similar, related, or noteworthy projects
  313. - [`usql`](https://github.com/xo/usql)
  314. - [`textql`](https://github.com/dinedal/textql)
  315. - [`golang-migrate`](https://github.com/golang-migrate/migrate)
  316. - [`octosql`](https://github.com/cube2222/octosql)
  317. - [`rq`](https://github.com/dflemstr/rq)
  318. - [`miller`](https://github.com/johnkerl/miller)
  319. - [`jsoncolor`](https://github.com/neilotoole/jsoncolor) is a JSON colorizer created for `sq`.
  320. - [`streamcache`](https://github.com/neilotoole/streamcache) is a Go in-memory byte cache
  321. mechanism created for `sq`.
  322. - [`fifomu`](https://github.com/neilotoole/fifomu) is a FIFO mutex, used by `streamcache`,
  323. and thus upstream in `sq`.
  324. - [`tailbuf`](https://github.com/neilotoole/tailbuf) is a fixed-size object tail buffer
  325. created for `sq`.
  326. - [`oncecache`](https://github.com/neilotoole/oncecache) is an in-memory object cache
  327. created for `sq`.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.