Skip to content
Back to search

fitter

New way for collect information from the API's/Websites

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/PxyUp/fitter with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/PxyUp/fitter'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/pxyup--fitter/badge.svg)](/listings/pxyup--fitter)

HTML

<a href="/listings/pxyup--fitter"><img src="/api/listings/pxyup--fitter/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Download `fitter-mcp-<os>-<arch>.mcpb` from the [release page](https://github.com/PxyUp/fitter/releases) and open it — Claude Desktop installs the server automatically.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 2,024 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Fitter — web data for AI agents
  2. [![MCP Toplist](https://mcptoplist.com/badge/io.github.PxyUp%2Ffitter.svg)](https://mcptoplist.com/server/io.github.PxyUp%2Ffitter)
  3. [![Release](https://img.shields.io/github/v/release/PxyUp/fitter)](https://github.com/PxyUp/fitter/releases)
  4. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/PxyUp/fitter/blob/master/LICENSE)
  5. [![Go Reference](https://pkg.go.dev/badge/github.com/PxyUp/fitter.svg)](https://pkg.go.dev/github.com/PxyUp/fitter)
  6. [![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2)](https://github.com/sponsors/PxyUp)
  7. **Fitter turns any website or API into structured JSON — declaratively.** One JSON/YAML config describes where the data lives (HTTP request, headless browser, file, static value) and what to extract (JSON paths, CSS selectors, XPath). No code, no brittle scraping scripts.
  8. **🚀 [Try it in your browser](https://pxyup.github.io/fitter/)** — the real engine compiled to WebAssembly: live examples, a visual config builder, no install.
  9. Because configs are plain data, **LLMs can author them**. The built-in [MCP server](#how-to-use-fitter_mcp) lets Claude Code, Claude Desktop, or any MCP client write and run scraping pipelines on your machine, on demand:
  10. > *"Get the top 5 HackerNews stories with titles and scores"* → the model authors a fitter config, validates it, runs it locally, and gets clean JSON back.
  11. **One engine, five ways to use it:**
  12. | | |
  13. |---|---|
  14. | 🤖 **Fitter MCP** | [MCP server](#how-to-use-fitter_mcp) exposing fitter to Claude Code, Claude Desktop and any MCP client |
  15. | 🧠 **Fitter Agent** | AI-powered CLI: natural language → config → executed result |
  16. | 🖥 **Fitter CLI** | run configs locally for test/debug/home usage |
  17. | 📦 **Fitter Lib** | embed the engine in your own Go program |
  18. | ⚙️ **Fitter** | long-running service mode with scheduling & notifications |
  19. **Why fitter for AI agents?**
  20. - **Declarative & auditable** — the agent produces a config you can read, save and re-run, not throwaway code
  21. - **Local-first** — all fetching happens on your machine; no third-party scraping API, no keys, no per-request billing
  22. - **Batteries included** — HTTP client, headless browser (Playwright/Chromium/Docker), JSON/HTML/XML/XPath/PDF parsing, pagination, cached references, host rate-limits — in a single static binary
  23. - **Reusable** — what the agent authored today becomes tomorrow's cron job or service config
  24. ![fitter demo — declarative config to structured JSON](https://github.com/PxyUp/fitter/blob/master/demo.svg?raw=true)
  25. # How to use Fitter_MCP
  26. Fitter MCP is a [Model Context Protocol](https://modelcontextprotocol.io) server (stdio transport) which lets any MCP client — Claude Code, Claude Desktop, IDE assistants, custom agents — run Fitter configs and get structured JSON back.
  27. ### Quick start (Claude Desktop — one click)
  28. Download `fitter-mcp-<os>-<arch>.mcpb` from the [release page](https://github.com/PxyUp/fitter/releases) and open it — Claude Desktop installs the server automatically.
  29. ### Quick start (Claude Code)
  30. ```bash
  31. # 1. get the binary: download fitter_mcp_<version>-<os>-<arch> from the release page
  32. # https://github.com/PxyUp/fitter/releases — or build it from source:
  33. go build -o fitter_mcp ./cmd/mcp
  34. # 2. register it once, available in every project
  35. claude mcp add fitter -s user -- "$(pwd)/fitter_mcp"
  36. ```
  37. Then just ask:
  38. > *Get the top 5 HackerNews stories with titles and scores using fitter*
  39. The model calls `fitter_config_reference`, authors a config, optionally checks it with `fitter_validate_config` and executes it via `fitter_run` — all data fetching happens locally on your machine. For a ready-made pipeline try [examples/config_morning_briefing.json](https://github.com/PxyUp/fitter/blob/master/examples/config_morning_briefing.json):
  40. > *Run examples/config_morning_briefing.json with fitter and give me the briefing*
  41. ### Register in Claude Desktop
  42. ```json
  43. {
  44. "mcpServers": {
  45. "fitter": {
  46. "command": "/path/to/fitter_mcp"
  47. }
  48. }
  49. }
  50. ```
  51. ### Browser support (Playwright)
  52. The `.mcpb` bundle and native binary ship **without** browsers: HTTP, static and file connectors work out of the box, but browser configs (the `playwright` connector) need Playwright's browsers. A few ways to get them:
  53. - **On first use (native binary / `.mcpb`):** set `"install": true` in the playwright connector — fitter downloads the driver + browser matching its built-in `playwright-go` version on first use (one-time, cached), so no separate install step is needed.
  54. - **Ahead of time (native, optional):** to avoid the first-run download, install the browsers beforehand with the same `playwright-go` version fitter is built against (check [`go.mod`](https://github.com/PxyUp/fitter/blob/master/go.mod), currently `v0.6100.0`):
  55. ```bash
  56. go run github.com/mxschmitt/playwright-go/cmd/playwright@v0.6100.0 install
  57. # Linux: append --with-deps to also install the required OS libraries
  58. ```
  59. The version must match `go.mod` exactly — `playwright-go` refuses to run against a mismatched driver. Then run configs without `"install": true`.
  60. - **Docker:** use the [`ghcr.io/pxyup/fitter-mcp:playwright`](#docker) image, which bundles Chromium, Firefox and WebKit preinstalled (no `"install": true` needed).
  61. ### Tools
  62. | Tool | Description |
  63. |------|-------------|
  64. | `fitter_run` | Run a Fitter config passed inline (JSON or YAML string) and return the extracted data as JSON. Accepts an optional `input` value available in the config via `{{{FromInput=.}}}` / `{{{FromInput=json.path}}}` |
  65. | `fitter_run_file` | Same as `fitter_run` but reads the config from a local `.json`/`.yaml` file |
  66. | `fitter_run_url` | Same as `fitter_run` but downloads the config from an HTTP(S) URL, e.g. a raw GitHub link |
  67. | `fitter_inspect_url` | Fetch a URL and return a compact structure outline + candidate selectors/paths (gjson paths for JSON; repeated-element/list-row selectors for HTML) so the model authors a config first-try instead of guessing selectors and getting nulls. Detects client-rendered SPAs and can `render` them in a headless browser. Read-only — does not extract |
  68. | `fitter_validate_config` | Validate a config without executing it (structure, `response_type`, connector data source, model). Useful while iterating on a config |
  69. | `fitter_config_reference` | Return a condensed reference of the whole config format (connectors, parsers, model/field schema, placeholders, notifiers, references, limits) with working examples, so the model can author configs without external docs |
  70. The reference is also exposed as MCP resource `fitter://config-reference` for clients that support resources.
  71. The config format is exactly the same as for [Fitter_CLI](#how-to-use-fitter_cli): a top-level object with `item` (required), `limits` and `references`. [Notifiers](#notifiers) work too (the result is additionally pushed to http/telegram/redis/file/console); `trigger_config` and `http_server` are service-mode only and are ignored in MCP calls.
  72. ### Remote / hosted mode (streamable HTTP)
  73. By default `fitter_mcp` talks stdio. Pass `--http` to serve the [streamable HTTP transport](https://modelcontextprotocol.io/docs/concepts/transports) instead — for a shared team server, a container, or any remote deployment:
  74. ```bash
  75. # serve MCP at http://<host>:8080/mcp (health probe at /healthz)
  76. FITTER_MCP_AUTH_TOKEN=my-secret fitter_mcp --http :8080
  77. # register the remote endpoint in Claude Code
  78. claude mcp add --transport http fitter http://localhost:8080/mcp --header "Authorization: Bearer my-secret"
  79. ```
  80. - `--http <addr>` (env `FITTER_MCP_HTTP_ADDR`) — listen address; stdio mode when empty
  81. - `FITTER_MCP_AUTH_TOKEN` — when set, every `/mcp` request must send `Authorization: Bearer <token>`; without it the endpoint is unauthenticated, so bind to localhost or put it behind a proxy
  82. - `--stateless` (env `FITTER_MCP_STATELESS=true`) — no per-session state, so replicas can sit behind a load balancer without sticky sessions
  83. - `--metrics-addr <addr>` (env `FITTER_MCP_METRICS_ADDR`) — serve Prometheus metrics on a separate address (e.g. `127.0.0.1:9091`); works in stdio mode too; empty disables. Exposes MCP request counts by method, tool call durations by tool and outcome, and outbound HTTP request durations by host, method and status. The endpoint is unauthenticated — bind it to localhost or a private interface
  84. The server shuts down gracefully on SIGINT/SIGTERM.
  85. #### Docker
  86. A slim multi-arch image (`linux/amd64` + `linux/arm64`) ships with every release:
  87. ```bash
  88. # hosted HTTP mode
  89. docker run --rm -p 8080:8080 \
  90. -e FITTER_MCP_HTTP_ADDR=:8080 \
  91. -e FITTER_MCP_AUTH_TOKEN=my-secret \
  92. ghcr.io/pxyup/fitter-mcp:latest
  93. # or stdio mode, spawned by the MCP client
  94. claude mcp add fitter -s user -- docker run --rm -i ghcr.io/pxyup/fitter-mcp:latest
  95. ```
  96. The slim image contains only the fitter binary and CA certificates: server/static/file connectors work, browser connectors (`chromium`/`docker`/`playwright`) do not.
  97. For browser-based configs use the `playwright` variant, which bundles Playwright with Chromium, Firefox and WebKit (matched to the `playwright-go` version fitter is built against, so no `"install": true` is needed in configs):
  98. ```bash
  99. docker run --rm -i ghcr.io/pxyup/fitter-mcp:playwright # stdio mode
  100. # per-release tag: ghcr.io/pxyup/fitter-mcp:vX.Y.Z-playwright
  101. ```
  102. It is built from [`Dockerfile.mcp-playwright`](https://github.com/PxyUp/fitter/blob/master/Dockerfile.mcp-playwright); build with `--build-arg PLAYWRIGHT_BROWSERS=chromium` for a smaller Chromium-only image.
  103. #### OAuth2 accounts in Docker
  104. Both images ship `fitter_cli`, so the one-time [OAuth2 login](#fitter_cli-auth--connect-an-oauth2-account) can run inside the container. Store the token on a volume mounted at `/tokens` (pre-created writable in the image) and share it with the MCP server:
  105. ```bash
  106. # one-time login, device flow: no ports needed — open the printed url on any device
  107. docker run --rm -it -v fitter-tokens:/tokens --entrypoint fitter_cli \
  108. ghcr.io/pxyup/fitter-mcp:latest \
  109. auth --provider github --client-id <ID> --client-secret <SECRET> --token-file /tokens/github.json
  110. # or browser flow (device flow not enabled for the app): publish the callback port and
  111. # bind on 0.0.0.0 so the published port reaches the listener; the browser still visits 127.0.0.1
  112. docker run --rm -it -p 8988:8988 -e FITTER_AUTH_LISTEN=0.0.0.0 \
  113. -v fitter-tokens:/tokens --entrypoint fitter_cli ghcr.io/pxyup/fitter-mcp:latest \
  114. auth --provider github --client-id <ID> --client-secret <SECRET> --token-file /tokens/github.json
  115. # then run the MCP server with the same volume; configs reference "token_file": "/tokens/github.json"
  116. # stdio mode (spawned by the MCP client, no port):
  117. docker run --rm -i -v fitter-tokens:/tokens ghcr.io/pxyup/fitter-mcp:latest
  118. # hosted HTTP mode (MCP endpoint on 8080, like the run examples above):
  119. docker run --rm -p 8080:8080 -v fitter-tokens:/tokens \
  120. -e FITTER_MCP_HTTP_ADDR=:8080 \
  121. -e FITTER_MCP_AUTH_TOKEN=my-secret \
  122. ghcr.io/pxyup/fitter-mcp:latest
  123. ```
  124. Note: `8988` is only for the one-time browser-flow login; the MCP server itself needs no port in stdio mode and only `8080` in hosted HTTP mode.
  125. #### Logged-in browser sessions in Docker
  126. [Browser sessions](#fitter_cli-browser-login--reuse-a-real-login-session) need the `playwright` image (the slim one has no browsers). The one-time headed login needs a display, so run it **on the host**, then bind-mount the session dir into the container (the image pre-creates a writable `/sessions`):
  127. ```bash
  128. # on the host: log in once, save the session
  129. fitter_cli browser-login --url https://example.com/login --storage-state ~/.fitter/sessions/example.json
  130. # run the MCP server with the sessions dir mounted; configs reference "storage_state_file": "/sessions/example.json"
  131. docker run --rm -i -v ~/.fitter/sessions:/sessions ghcr.io/pxyup/fitter-mcp:playwright
  132. ```
  133. Use a bind mount (not a named volume): the container writes refreshed cookies back after every run, so the host copy stays current and can be re-extended with `browser-login` at any time.
  134. The volume must stay writable for the server: rotated refresh tokens are written back on every refresh.
  135. ### Environment variables
  136. 1. **FITTER_PLUGINS** - string[""] - [path for plugins folder](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md), same as the `--plugins` flag of Fitter/Fitter_CLI
  137. 2. **FITTER_MCP_HTTP_ADDR** - string[""] - listen address for [remote mode](#remote--hosted-mode-streamable-http), same as `--http`
  138. 3. **FITTER_MCP_AUTH_TOKEN** - string[""] - bearer token protecting the HTTP endpoint
  139. 4. **FITTER_MCP_STATELESS** - bool[false] - stateless HTTP transport, same as `--stateless`
  140. 5. **FITTER_MCP_METRICS_ADDR** - string[""] - Prometheus metrics listen address, same as `--metrics-addr`
  141. # Recipes
  142. Complete, tested configs showing the main patterns. All of them run unchanged via [Fitter_MCP](#how-to-use-fitter_mcp) (`fitter_run_file`), [Fitter_CLI](#how-to-use-fitter_cli) or the [library](#use-like-a-library) — more in [examples/](https://github.com/PxyUp/fitter/tree/master/examples).
  143. ### Scrape a page that has no API, enrich from one that does
  144. GitHub trending has no official API — scrape the HTML for repo slugs (`html_attribute` reads the `href`), then fan each one out into the GitHub REST API with `{PL}`:
  145. [examples/config_github_trending.json](https://github.com/PxyUp/fitter/blob/master/examples/config_github_trending.json)
  146. ```json
  147. {
  148. "item": {
  149. "connector_config": {
  150. "response_type": "HTML",
  151. "url": "https://github.com/trending",
  152. "server_config": { "method": "GET", "headers": { "User-Agent": "Mozilla/5.0 (fitter demo)" } }
  153. },
  154. "model": {
  155. "array_config": {
  156. "root_path": "article.Box-row h2 a",
  157. "length_limit": 5,
  158. "item_config": {
  159. "field": {
  160. "type": "string",
  161. "html_attribute": "href",
  162. "generated": { "model": {
  163. "type": "object",
  164. "connector_config": {
  165. "response_type": "json",
  166. "url": "https://api.github.com/repos{PL}",
  167. "server_config": { "method": "GET", "headers": { "User-Agent": "fitter-demo" } },
  168. "null_on_error": true
  169. },
  170. "model": { "object_config": { "fields": {
  171. "repo": { "base_field": { "type": "string", "path": "full_name" } },
  172. "stars": { "base_field": { "type": "int", "path": "stargazers_count" } },
  173. "language": { "base_field": { "type": "string", "path": "language" } }
  174. } } }
  175. } }
  176. }
  177. }
  178. }
  179. }
  180. },
  181. "limits": { "host_request_limiter": { "api.github.com": 2 } }
  182. }
  183. ```
  184. ```json
  185. [{"repo": "block/buzz", "stars": 6214, "language": "Rust"}, {"repo": "koala73/worldmonitor", "stars": 71179, "language": "TypeScript"}]
  186. ```
  187. ### Join on a JSON field with an expression
  188. When array items are objects, the join key lives inside them — pull it out with `{{{FromExp=...}}}` ([expr-lang](https://expr-lang.org/) over `fRes`, the current item). Book search → author details, search query supplied via `input`:
  189. [examples/config_book_authors.json](https://github.com/PxyUp/fitter/blob/master/examples/config_book_authors.json)
  190. ```json
  191. "url": "https://openlibrary.org/authors/{{{FromExp=fromJSON(fRes).author_key[0]}}}.json"
  192. ```
  193. ```bash
  194. ./fitter_cli --path=examples/config_book_authors.json --input=dune
  195. ```
  196. ```json
  197. [{"title": "Dune", "year": 1965, "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}}]
  198. ```
  199. ### Write results to a local file
  200. The [file_storage](#file-storage-field) generated field turns fields into writes — top-5 crypto coins appended to a CSV, one row per item. Bare `{{{json.path}}}` placeholders read the current item; `{HUMAN_INDEX}` stamps the 1-based rank (items are processed in parallel, so appends land in completion order — sort by the rank column):
  201. [examples/config_crypto_csv.json](https://github.com/PxyUp/fitter/blob/master/examples/config_crypto_csv.json)
  202. ```json
  203. "file_storage": {
  204. "content": "{HUMAN_INDEX},{{{name}}},{{{current_price}}},{{{price_change_percentage_24h}}}\n",
  205. "file_name": "coins.csv",
  206. "path": "/tmp/fitter-report",
  207. "append": true
  208. }
  209. ```
  210. ```bash
  211. $ sort -n /tmp/fitter-report/coins.csv
  212. 1,Bitcoin,64778,-2.3
  213. 2,Ethereum,1881.01,-3.4
  214. 3,Tether,0.999265,0
  215. ```
  216. ### Extract text from a PDF
  217. `response_type: "pdf"` turns any fetched PDF into a JSON document — `{"text": "...", "pages": ["..."], "total_pages": N}` — so regular JSON paths (`text`, `pages.0`) and expressions work on it. The Bitcoin whitepaper, page count plus a trimmed intro:
  218. [examples/config_pdf.json](https://github.com/PxyUp/fitter/blob/master/examples/config_pdf.json)
  219. ```json
  220. {
  221. "item": {
  222. "connector_config": {
  223. "response_type": "pdf",
  224. "url": "https://bitcoin.org/bitcoin.pdf",
  225. "server_config": { "method": "GET" }
  226. },
  227. "model": {
  228. "object_config": {
  229. "fields": {
  230. "total_pages": { "base_field": { "type": "int", "path": "total_pages" } },
  231. "intro": {
  232. "base_field": {
  233. "type": "string",
  234. "path": "pages.0",
  235. "generated": {
  236. "calculated": {
  237. "type": "string",
  238. "expression": "trim(fRes[:100]) + \"...\""
  239. }
  240. }
  241. }
  242. }
  243. }
  244. }
  245. }
  246. }
  247. }
  248. ```
  249. ```json
  250. {"intro": "Bitcoin: A Peer-to-Peer Electronic Cash SystemSatoshi Nakamotosatoshin@gmx.comwww.bitcoin.orgAbstrac...", "total_pages": 9}
  251. ```
  252. # Way to collect information
  253. 1. **Server** - parsing response from some API's or http request(usage of http.Client)
  254. 2. **Browser** - emulate real browser using chromium + docker + playwright/cypress and get DOM information
  255. 3. **Static** - parsing static string as data
  256. # Format which can be parsed
  257. 1. **JSON** - parsing JSON to get specific information
  258. 2. **XML** - parsing xml tree to get specific information
  259. 3. **HTML** - parsing dom tree to get specific information
  260. 4. **XPath** - parsing dom tree to get specific information but by xpath
  261. 5. **PDF** - extracting text from PDF documents; the content is exposed as JSON `{"text": "...", "pages": ["..."], "total_pages": N}` so regular JSON paths like `text` or `pages.0` work
  262. # Use like a library
  263. ```bash
  264. go get github.com/PxyUp/fitter
  265. ```
  266. ```go
  267. package main
  268. import (
  269. "fmt"
  270. "github.com/PxyUp/fitter/lib"
  271. "github.com/PxyUp/fitter/pkg/config"
  272. "log"
  273. "net/http"
  274. )
  275. func main() {
  276. res, err := lib.Parse(&config.Item{
  277. ConnectorConfig: &config.ConnectorConfig{
  278. ResponseType: config.Json,
  279. Url: "https://random-data-api.com/api/appliance/random_appliance",
  280. ServerConfig: &config.ServerConnectorConfig{
  281. Method: http.MethodGet,
  282. },
  283. },
  284. Model: &config.Model{
  285. ObjectConfig: &config.ObjectConfig{
  286. Fields: map[string]*config.Field{
  287. "my_id": {
  288. BaseField: &config.BaseField{
  289. Type: config.Int,
  290. Path: "id",
  291. },
  292. },
  293. "generated_id": {
  294. BaseField: &config.BaseField{
  295. Generated: &config.GeneratedFieldConfig{
  296. UUID: &config.UUIDGeneratedFieldConfig{},
  297. },
  298. },
  299. },
  300. "generated_array": {
  301. ArrayConfig: &config.ArrayConfig{
  302. RootPath: "@this|@keys",
  303. ItemConfig: &config.ObjectConfig{
  304. Field: &config.BaseField{
  305. Type: config.String,
  306. },
  307. },
  308. },
  309. },
  310. },
  311. },
  312. },
  313. }, nil, nil, nil, nil)
  314. if err != nil {
  315. log.Fatal(err)
  316. }
  317. fmt.Println(res.ToJson())
  318. }
  319. ```
  320. Output:
  321. ```json
  322. {
  323. "generated_array": ["id","uid","brand","equipment"],
  324. "my_id": 6000,
  325. "generated_id": "26b08b73-2f2e-444d-bcf2-dac77ac3130e"
  326. }
  327. ```
  328. Use `lib.ParseCtx(ctx, ...)` to pass a `context.Context`: cancelling it aborts in-flight fetches (HTTP requests, headless browsers, docker containers) and applies deadlines end-to-end. `lib.Parse` is equivalent to `lib.ParseCtx(context.Background(), ...)`.
  329. # How to use Fitter
  330. [Download latest version from the release page](https://github.com/PxyUp/fitter/releases)
  331. or locally:
  332. ```bash
  333. go run cmd/fitter/main.go --path=./examples/config_api.json
  334. ```
  335. ### Arguments
  336. 1. **--path** - string[""] - path for the configuration of the Fitter
  337. 2. **--url** - string[""] - url for the configuration of the Fitter
  338. 3. **--verbose** - bool[false] - enable logging
  339. 4. **--plugins** - string[""] - [path for plugins for Fitter](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md)
  340. 5. **--log-level** - enum["info", "error", "debug", "fatal"] - set log level(only if verbose set to true)
  341. # How to use Fitter_CLI
  342. [Download latest version from the release page](https://github.com/PxyUp/fitter/releases)
  343. or locally:
  344. ```bash
  345. go run cmd/cli/main.go --path=./examples/cli/config_cli.json
  346. ```
  347. ### Arguments
  348. 1. **--path** - string[""] - path for the configuration of the Fitter_CLI
  349. 2. **--url** - string[""] - url for the configuration of the Fitter_CLI
  350. 3. **--copy** - bool[false] - copy information into clipboard
  351. 4. **--pretty** - bool[true] - make readable result(also affect on copy)
  352. 5. **--verbose** - bool[false] - enable logging
  353. 6. **--omit-error-pretty** - bool[false] - Provide pure value if pretty is invalid
  354. 7. **--plugins** - string[""] - [path for plugins for Fitter](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md)
  355. 8. **--log-level** - enum["info", "error", "debug", "fatal"] - set log level(only if verbose set to true)
  356. 9. **--input** - string[""] - specify input value for [formatting](#placeholder-list). Examples: `--input=\""124"\"` `--input=124` `--input='{"test": 5}'`
  357. ```bash
  358. ./fitter_cli_${VERSION} --path=./examples/cli/config_cli.json --copy=true
  359. ```
  360. ### fitter_cli auth — connect an OAuth2 account
  361. One-time interactive login which stores a (refresh) token for the [oauth2 connector config](#oauth2-config):
  362. ```bash
  363. # device flow (default when the provider supports it): no callback, works headless
  364. ./fitter_cli_${VERSION} auth --provider github --client-id <ID> --client-secret <SECRET> --token-file ~/.fitter/tokens/github.json
  365. # custom provider without preset
  366. ./fitter_cli_${VERSION} auth --auth-url https://.../authorize --token-url https://.../token --client-id <ID> --token-file ./token.json
  367. ```
  368. Arguments:
  369. 1. **--provider** - preset with known endpoints: `github|google|microsoft|gitlab|spotify`
  370. 2. **--client-id** / **--client-secret** - OAuth2 app credentials (some device flows work without secret)
  371. 3. **--token-file** - where to store the received token (0600 permissions); reference the same path in `oauth2.token_file`
  372. 4. **--flow** - `auto` (device if available, else browser), `device` (visit a url + enter a code) or `browser` (localhost callback with PKCE, default port 8988 — register `http://127.0.0.1:8988/callback` as the app callback url)
  373. 5. **--scopes** - comma separated scopes
  374. 6. **--auth-url/--token-url/--device-auth-url/--auth-style** - endpoint overrides for providers without preset
  375. 7. **--port** - int[8988] - browser flow callback port (env `FITTER_AUTH_PORT`); with the default the callback url to register at the provider is `http://127.0.0.1:8988/callback`
  376. 8. **--listen** - browser flow bind address, default `127.0.0.1`; set `0.0.0.0` inside a container so the published port reaches the listener (env `FITTER_AUTH_LISTEN`)
  377. 9. **--redirect-url** - callback url registered at the provider when it differs from the listen address, e.g. docker port mapping (env `FITTER_AUTH_REDIRECT_URL`)
  378. 10. **--no-browser** - only print the authorization url
  379. Running inside Docker: see [OAuth2 accounts in Docker](#oauth2-accounts-in-docker).
  380. After login the command prints the ready-to-use `oauth2` config block. The connector refreshes the access token automatically and writes rotated refresh tokens back to the token file, so the login is needed only once.
  381. ### fitter_cli browser-login — reuse a real login session
  382. For sites without an API/OAuth: log in once by hand in a real (headed) browser window — any auth scheme works, including passwords, 2FA, SSO and captchas — and save the session for headless scraping via [storage_state_file](#playwright):
  383. ```bash
  384. ./fitter_cli_${VERSION} browser-login --url https://example.com/login --storage-state ~/.fitter/sessions/example.json
  385. # a browser window opens; log in, then press Enter in the terminal to save the session
  386. ```
  387. Arguments:
  388. 1. **--url** - login page to open (required)
  389. 2. **--storage-state** - where to save the session (cookies + localStorage, 0600 permissions); reference the same path in `playwright.storage_state_file` (required)
  390. 3. **--browser** - enum["Chromium", "FireFox", "WebKit"] default "Chromium"; use the same value as the scraping config — sites may bind sessions to the browser fingerprint
  391. 4. **--install** - bool[false] - install playwright browsers first
  392. 5. **--indexeddb** - bool[false] - include IndexedDB in the snapshot (Firebase Auth and similar)
  393. Re-running the command loads the existing state first, so you can extend/refresh a session without logging in from scratch. The scraping connector also writes refreshed cookies back after every run, keeping the session alive as long as it is used regularly. Needs a display: inside Docker run this command on the host and mount the file — see [browser sessions in Docker](#logged-in-browser-sessions-in-docker).
  394. Examples:
  395. 1. **Server version** [HackerNews + Quotes + Guardian News](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json) - using API + HTML + XPath parsing
  396. 2. **Chromium version** [Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_browser.json) - using HTML parsing + browser emulation
  397. 3. **Docker version** [Docker version: Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_docker.json) - using HTML parsing + browser from Docker image
  398. 4. **Playwright version** [Playwright version: Guardian News + Quotes](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_playwright.json) - using HTML parsing + browser from Playwright framework
  399. 5. **Playwright version** [Playwright version: England Cities + Weather](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_weather.json) - using HTML + XPath parsing + browser from Playwright framework
  400. 6. **JSON version** [Generate pagination](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_static_connector.json) - using static connector for generate pagination array
  401. 7. **Server version** [Get current time](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_current_time.json) - get time from url and format it
  402. # How to use Fitter_Agent
  403. Fitter Agent is an AI-powered CLI that uses Claude to convert natural language requests into Fitter configurations and execute them automatically.
  404. [Download latest version from the release page](https://github.com/PxyUp/fitter/releases)
  405. or locally:
  406. ```bash
  407. export ANTHROPIC_API_KEY=<your-anthropic-api-key>
  408. go run cmd/agent/main.go
  409. ```
  410. ### Arguments
  411. 1. **--api-key** - string[""] - Anthropic API key. Prefer the `ANTHROPIC_API_KEY` environment variable so the key does not end up in your shell history
  412. 2. **--model** - string["claude-opus-4-8"] - Claude model to use
  413. 3. **--effort** - enum["low", "medium", "high", "xhigh", "max"] - reasoning effort, default "high". Lower it for faster/cheaper configs, raise it for harder extractions
  414. 4. **--verbose** - bool[false] - enable logging
  415. 5. **--log-level** - enum["info", "error", "debug", "fatal"] - set log level
  416. 6. **--plugins** - string[""] - path for plugins for Fitter
  417. 7. **--chromium-limit** - uint[0] - limit concurrent Chromium instances
  418. 8. **--docker-limit** - uint[0] - limit concurrent Docker containers
  419. 9. **--playwright-limit** - uint[0] - limit concurrent Playwright instances
  420. ### How it works
  421. ```
  422. ┌─────────────────────────────────────────────────────────────────┐
  423. │ 1. User enters natural language request │
  424. │ "Get top 5 HackerNews stories with titles and scores" │
  425. │ ↓ │
  426. │ 2. Claude returns a config in a schema-constrained response │
  427. │ ↓ │
  428. │ 3. Agent validates it; on failure the error is handed back │
  429. │ to Claude to repair (up to 3 attempts) │
  430. │ ↓ │
  431. │ 4. Agent displays config and asks for confirmation │
  432. │ ↓ │
  433. │ 5. On confirmation, executes via lib.Parse() │
  434. │ ↓ │
  435. │ 6. Returns structured JSON result │
  436. └─────────────────────────────────────────────────────────────────┘
  437. ```
  438. ### Refining a config
  439. The agent keeps the conversation, so after a config is generated you can just
  440. say what to change instead of restating the whole request:
  441. ```
  442. > Get top 3 HackerNews stories with titles and scores
  443. refine> Only return 5 items and also include the article URL
  444. ```
  445. Use **new** to forget the current config and start a fresh session.
  446. ### Interactive REPL Commands
  447. - **help** - show help message
  448. - **new/reset** - forget the current config and start fresh
  449. - **clear** - clear the screen
  450. - **exit/quit/q** - exit the agent
  451. ### Example Session
  452. ```bash
  453. $ export ANTHROPIC_API_KEY=sk-ant-...
  454. $ ./fitter_agent
  455. ╔══════════════════════════════════════════════════════════════╗
  456. ║ Fitter Agent - AI-Powered Data Extraction ║
  457. ╚══════════════════════════════════════════════════════════════╝
  458. Describe what you want to extract. Follow-up messages refine the
  459. previous config. Type 'help' for commands.
  460. > Get top 3 HackerNews stories with titles and scores
  461. ┌─ Generated Fitter Config ───────────────────────────────────────
  462. {
  463. "item": {
  464. "connector_config": {
  465. "response_type": "json",
  466. "url": "https://hacker-news.firebaseio.com/v0/topstories.json",
  467. "server_config": { "method": "GET" }
  468. },
  469. "model": {
  470. "array_config": {
  471. "root_path": "@this",
  472. "length_limit": 3,
  473. "item_config": {
  474. "fields": {
  475. "id": { "base_field": { "type": "int" } },
  476. "story": {
  477. "base_field": {
  478. "type": "int",
  479. "generated": {
  480. "model": {
  481. "type": "object",
  482. "connector_config": {
  483. "response_type": "json",
  484. "url": "https://hacker-news.firebaseio.com/v0/item/{PL}.json",
  485. "server_config": { "method": "GET" }
  486. },
  487. "model": {
  488. "object_config": {
  489. "fields": {
  490. "title": { "base_field": { "type": "string", "path": "title" } },
  491. "score": { "base_field": { "type": "int", "path": "score" } }
  492. }
  493. }
  494. }
  495. }
  496. }
  497. }
  498. }
  499. }
  500. }
  501. }
  502. }
  503. }
  504. }
  505. └──────────────────────────────────────────────────────────────────
  506. Execute this config? [y/n]: y
  507. ┌─ Result ────────────────────────────────────────────────────────
  508. [
  509. {
  510. "id": 46740029,
  511. "story": { "title": "Show HN: Open-source project", "score": 161 }
  512. },
  513. {
  514. "id": 46737630,
  515. "story": { "title": "Interesting article", "score": 237 }
  516. },
  517. {
  518. "id": 46735644,
  519. "story": { "title": "New technology release", "score": 192 }
  520. }
  521. ]
  522. └──────────────────────────────────────────────────────────────────
  523. > exit
  524. Goodbye!
  525. ```
  526. ### Example Requests
  527. | Request | What it does |
  528. |---------|--------------|
  529. | `Get Bitcoin price from CoinGecko API` | Fetches current BTC price |
  530. | `Scrape headlines from news.ycombinator.com with links` | HTML scraping with CSS selectors |
  531. | `Get top 5 stories from HackerNews with titles` | Nested API calls |
  532. | `Fetch weather data from wttr.in for London` | Simple API extraction |
  533. | `Scrape product names and prices from example.com` | Web scraping |
  534. ### Supported Capabilities
  535. The agent can generate configs for:
  536. - **JSON APIs** - REST APIs with GET/POST methods
  537. - **HTML Scraping** - CSS selector-based extraction
  538. - **XPath Scraping** - XPath-based extraction
  539. - **Nested API Calls** - Fetch details for each item in a list
  540. - **Browser Emulation** - Playwright for JS-rendered pages
  541. - **Formatted Fields** - URL templates with placeholders
  542. - **Array Limiting** - Limit results to N items
  543. # Configuration
  544. ## Connector
  545. It is the way how you fetch the data
  546. ```go
  547. type ConnectorConfig struct {
  548. ResponseType ParserType `json:"response_type" yaml:"response_type"`
  549. Url string `json:"url" yaml:"url"`
  550. Attempts uint32 `json:"attempts" yaml:"attempts"`
  551. NullOnError bool `yaml:"null_on_error" json:"null_on_error"`
  552. StaticConfig *StaticConnectorConfig `json:"static_config" yaml:"static_config"`
  553. IntSequenceConfig *IntSequenceConnectorConfig `json:"int_sequence_config" yaml:"int_sequence_config"`
  554. ServerConfig *ServerConnectorConfig `json:"server_config" yaml:"server_config"`
  555. BrowserConfig *BrowserConnectorConfig `yaml:"browser_config" json:"browser_config"`
  556. PluginConnectorConfig *PluginConnectorConfig `json:"plugin_connector_config" yaml:"plugin_connector_config"`
  557. ReferenceConfig *ReferenceConnectorConfig `yaml:"reference_config" json:"reference_config"`
  558. FileConfig *FileConnectorConfig `json:"file_config" yaml:"file_config"`
  559. }
  560. ```
  561. - NullOnError[false] - if set to true then all errors a ignored
  562. - ResponseType - enum["HTML", "json", "xpath", "XML", "pdf"] - in which format data comes from the connector
  563. - Attempts - how many attempts to use for fetch data by connector
  564. - Url - define which address to request. Important: can be with [inject of the parent value as a string](#placeholder-list)
  565. `https://api.open-meteo.com/v1/forecast?latitude={{{latitude}}}&longitude={{{longitude}}}&hourly=temperature_2m&forecast_days=1`
  566. Config can be one of:
  567. - [ServerConfig](#serverconnectorconfig)
  568. - [BrowserConfig](#browserconnectorconfig)
  569. - [StaticConfig](#staticconnectorconfig)
  570. - [PluginConnectorConfig](#pluginconnectorconfig)
  571. - [ReferenceConfig](#referenceconnectorconfig)
  572. - [IntSequenceConfig](#intsequenceconnectorconfig)
  573. - [FileConfig](#fileconnectorconfig)
  574. Example:
  575. ```json
  576. {
  577. "response_type": "xpath",
  578. "attempts": 3,
  579. "url": "https://openweathermap.org/find?q={PL}",
  580. "browser_config": {
  581. "playwright": {
  582. "timeout": 30,
  583. "wait": 30,
  584. "install": false,
  585. "browser": "Chromium"
  586. }
  587. }
  588. }
  589. ```
  590. ### PluginConnectorConfig
  591. Connector can be defined via plugin system. For use that you need apply next flags to Fitter/Cli(location of the plugins):
  592. ```bash
  593. ... --plugins=./examples/plugin
  594. ```
  595. --plugins - looking for all files with ".so" extension in provided folder(subdirs excluded)
  596. ```go
  597. type PluginConnectorConfig struct {
  598. Name string `json:"name" yaml:"name"`
  599. Config json.RawMessage `json:"config" yaml:"config"`
  600. }
  601. ```
  602. ```json
  603. {
  604. "name": "connector",
  605. "config": {
  606. "name": "Elon"
  607. }
  608. }
  609. ```
  610. - Name - name of the plugin
  611. - Config - json config of the plugin
  612. #### How to build plugin
  613. Build plugin
  614. ```bash
  615. go build -buildmode=plugin -gcflags="all=-N -l" -o examples/plugin/connector.so examples/plugin/connector/connector.go
  616. ```
  617. Make sure you export **Plugin** variable which implements **pl.ConnectorPlugin** interface
  618. Example for CLI:
  619. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_plugin.json#L5
  620. Plugin example:
  621. ```go
  622. package main
  623. import (
  624. "context"
  625. "encoding/json"
  626. "fmt"
  627. "github.com/PxyUp/fitter/pkg/config"
  628. "github.com/PxyUp/fitter/pkg/logger"
  629. "github.com/PxyUp/fitter/pkg/builder"
  630. pl "github.com/PxyUp/fitter/pkg/plugins/plugin"
  631. )
  632. var (
  633. _ pl.ConnectorPlugin = &plugin{}
  634. Plugin plugin
  635. )
  636. type plugin struct {
  637. log logger.Logger
  638. Name string `json:"name" yaml:"name"`
  639. }
  640. func (pl *plugin) Get(ctx context.Context, parsedValue builder.Interfacable, index *uint32, input builder.Interfacable) ([]byte, error) {
  641. return []byte(fmt.Sprintf(`{"name": "%s"}`, pl.Name)), nil
  642. }
  643. func (pl *plugin) SetConfig(cfg *config.PluginConnectorConfig, logger logger.Logger) {
  644. pl.log = logger
  645. if cfg.Config != nil {
  646. err := json.Unmarshal(cfg.Config, pl)
  647. if err != nil {
  648. pl.log.Errorw("cant unmarshal plugin configuration", "error", err.Error())
  649. return
  650. }
  651. }
  652. }
  653. ```
  654. ### ReferenceConnectorConfig
  655. Connector which allow get prefetched data from [references](#references)
  656. ```go
  657. type ReferenceConnectorConfig struct {
  658. Name string `yaml:"name" json:"name"`
  659. }
  660. ```
  661. Example
  662. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L66
  663. - Name - reference name from [references](#references) map
  664. ### IntSequenceConnectorConfig
  665. Improved version of static connector which generate int sequence as result
  666. ```go
  667. type IntSequenceConnectorConfig struct {
  668. Start int `json:"start" yaml:"start"`
  669. End int `json:"end" yaml:"end"`
  670. Step int `json:"step" yaml:"step"`
  671. }
  672. ```
  673. - Start[0] - start point for generation(**included**)
  674. - End[0] - end point for generation(**excluded** from final result like range in any lang)
  675. - Step[1] - interval for sequence
  676. Example
  677. ```json
  678. {
  679. "start": 0,
  680. "end": 2
  681. // Generate [0, 1]
  682. }
  683. ```
  684. [Config example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_seq.json)
  685. ### FileConnectorConfig
  686. Connector type which fetch data from provided file
  687. ```go
  688. type FileConnectorConfig struct {
  689. Path string `yaml:"path" json:"path"`
  690. UseFormatting bool `yaml:"use_formatting" json:"use_formatting"`
  691. }
  692. ```
  693. - Path - file path. Support [formatting](#placeholder-list)
  694. - UseFormatting[false] - use [formatting](#placeholder-list) file content or not
  695. ### StaticConnectorConfig
  696. Connector type which fetch data from provided string
  697. ```go
  698. type StaticConnectorConfig struct {
  699. Value string `json:"value" yaml:"value"`
  700. Raw json.RawMessage `json:"raw" yaml:"raw"`
  701. }
  702. ```
  703. - Value - static string as data, can be html, json
  704. - Raw - accept raw json. [Example](https://github.com/PxyUp/fitter/blob/master/examples/config_static.json). Also support [formatting](#placeholder-list)
  705. Example:
  706. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_static_connector.json#L5
  707. ```json
  708. {
  709. "value": "[1,2,3,4,5]"
  710. }
  711. ```
  712. ### ServerConnectorConfig
  713. Connector type which fetch data using golang http.Client(server side request like curl)
  714. ```go
  715. type ServerConnectorConfig struct {
  716. Method string `json:"method" yaml:"method"`
  717. Headers map[string]string `yaml:"headers" json:"headers"`
  718. Timeout uint32 `yaml:"timeout" json:"timeout"`
  719. JsonRawBody json.RawMessage `json:"json_raw_body" yaml:"json_raw_body"`
  720. Body string `yaml:"body" json:"body"`
  721. ErrorOnStatus bool `json:"error_on_status" yaml:"error_on_status"`
  722. Proxy *ProxyConfig `yaml:"proxy" json:"proxy"`
  723. OAuth2 *OAuth2Config `yaml:"oauth2" json:"oauth2"`
  724. }
  725. ```
  726. - Method - supported all http methods: GET, POST, PUT, DELETE, PATCH, OPTIONS, HEAD
  727. - Headers - predefine headers for using during request [can be injected into key/value](#placeholder-list)
  728. - Timeout[sec] - default 60sec timeout or used provided
  729. - Body - body of the request, parsed value [can be injected](#placeholder-list)
  730. - JsonRawBody - body of the request in json format; value [can be injected](#placeholder-list)
  731. - ErrorOnStatus - optional, default `false`. When `true`, an HTTP response status `>= 400` is treated as a fetch error (flowing through `attempts` / `null_on_error`) instead of parsing the error body — so you can distinguish a failed fetch from a genuinely empty result. Leaving it `false` keeps the original behavior of parsing whatever body came back.
  732. - Proxy - setup proxy for request [config](#proxy-config)
  733. - OAuth2 - fetch/refresh an access token automatically and send it as `Authorization` header [config](#oauth2-config)
  734. > Requests send an identifiable `User-Agent` (`fitter (+https://github.com/PxyUp/fitter)`) by default; set your own `User-Agent` in `Headers` to override it.
  735. Example:
  736. ```json
  737. {
  738. "method": "GET",
  739. "proxy": {
  740. "server": "http://localhost:8080",
  741. "username": "pyx"
  742. }
  743. }
  744. ```
  745. ##### OAuth2 config
  746. Automatically obtains an access token before the request and injects it as the `Authorization` header (overriding one set via `headers`). Tokens are cached in memory and refreshed before expiry; on a `401` response the cached token is dropped and the request is retried once with a fresh one.
  747. ```go
  748. type OAuth2Config struct {
  749. TokenUrl string `json:"token_url" yaml:"token_url"`
  750. GrantType OAuth2GrantType `json:"grant_type" yaml:"grant_type"`
  751. ClientId string `json:"client_id" yaml:"client_id"`
  752. ClientSecret string `json:"client_secret" yaml:"client_secret"`
  753. Scopes []string `json:"scopes" yaml:"scopes"`
  754. RefreshToken string `json:"refresh_token" yaml:"refresh_token"`
  755. EndpointParams map[string]string `json:"endpoint_params" yaml:"endpoint_params"`
  756. AuthStyle string `json:"auth_style" yaml:"auth_style"`
  757. TokenFile string `json:"token_file" yaml:"token_file"`
  758. }
  759. ```
  760. - TokenUrl - token endpoint URL. Also support [formatting](#placeholder-list)
  761. - GrantType - enum["client_credentials", "refresh_token"], default is "client_credentials". Use "refresh_token" for APIs where the user consented once (Google, Microsoft, ...) and you hold a long-lived refresh token
  762. - ClientId/ClientSecret - client credentials. Also support [formatting](#placeholder-list), e.g. `{{{FromEnv=CLIENT_SECRET}}}`
  763. - Scopes - requested scopes
  764. - RefreshToken - required for the "refresh_token" grant. Also support [formatting](#placeholder-list)
  765. - EndpointParams - extra token endpoint parameters (e.g. `audience` for Auth0), "client_credentials" grant only
  766. - AuthStyle - enum["", "header", "params"] - how client credentials are passed to the token endpoint: basic auth header or request body; empty means auto detect
  767. - TokenFile - optional path (supports `~/`) for persisting tokens between runs; the stored token is preferred over RefreshToken and rotated refresh tokens are written back — required for providers with single-use refresh tokens (GitHub Apps and similar). Create it with [fitter_cli auth](#fitter_cli-auth--connect-an-oauth2-account)
  768. Example:
  769. ```json
  770. {
  771. "method": "GET",
  772. "oauth2": {
  773. "token_url": "https://oauth2.googleapis.com/token",
  774. "grant_type": "refresh_token",
  775. "client_id": "{{{FromEnv=GOOGLE_CLIENT_ID}}}",
  776. "client_secret": "{{{FromEnv=GOOGLE_CLIENT_SECRET}}}",
  777. "refresh_token": "{{{FromEnv=GOOGLE_REFRESH_TOKEN}}}"
  778. }
  779. }
  780. ```
  781. ##### Proxy config
  782. ```go
  783. type ProxyConfig struct {
  784. // Proxy to be used for all requests. HTTP and SOCKS proxies are supported, for example
  785. // `http://myproxy.com:3128` or `socks5://myproxy.com:3128`. Short form `myproxy.com:3128`
  786. // is considered an HTTP proxy.
  787. Server string `json:"server" yaml:"server"`
  788. // Optional username to use if HTTP proxy requires authentication.
  789. Username string `json:"username" yaml:"username"`
  790. // Optional password to use if HTTP proxy requires authentication.
  791. Password string `json:"password" yaml:"password"`
  792. }
  793. ```
  794. - Server - address with schema of proxy server. Also support [formatting](#placeholder-list)
  795. - Username - username for proxy(can be empty). Also support [formatting](#placeholder-list)
  796. - Password - password for proxy(can be empty). Also support [formatting](#placeholder-list)
  797. ```json
  798. {
  799. "server": "http://localhost:8080",
  800. "username": "pyx"
  801. }
  802. ```
  803. ##### Environment variables
  804. 1. **FITTER_HTTP_WORKER** - int[1000] - default concurrent HTTP workers
  805. ### BrowserConnectorConfig
  806. Connector type which emulate fetching of data via browser
  807. ```go
  808. type BrowserConnectorConfig struct {
  809. Chromium *ChromiumConfig `json:"chromium" yaml:"chromium"`
  810. Docker *DockerConfig `json:"docker" yaml:"docker"`
  811. Playwright *PlaywrightConfig `json:"playwright" yaml:"playwright"`
  812. }
  813. ```
  814. Config can be one of:
  815. - [Chromium](#chromium) - use local installed Chromium for fetch data
  816. - [Docker](#docker) - use docker as service for spin up container for fetch data
  817. - [Playwright](#playwright) - use playwright framework for fetch data
  818. Example:
  819. ```json
  820. {
  821. "docker": {
  822. "wait": 10000,
  823. "image": "docker.io/zenika/alpine-chrome:with-node",
  824. "entry_point": "chromium-browser",
  825. "purge": true
  826. }
  827. }
  828. ```
  829. #### Chromium
  830. Use locally installed Chromium for fetch the data
  831. ```go
  832. type ChromiumConfig struct {
  833. Path string `yaml:"path" json:"path"`
  834. Timeout uint32 `yaml:"timeout" json:"timeout"`
  835. Wait uint32 `yaml:"wait" json:"wait"`
  836. Flags []string `yaml:"flags" json:"flags"`
  837. }
  838. ```
  839. - Path - path to binary of Chromium
  840. - Timeout[sec] - timeout for execution of the chromium
  841. - Wait[msec] - timeout of page loading
  842. - Flags - flags for Chromium default: "--headless", "--proxy-auto-detect", "--temp-profile", "--incognito", "--disable-logging", "--disable-extensions", "--no-sandbox"
  843. Example:
  844. ```json
  845. {
  846. "path": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
  847. "wait": 10000
  848. }
  849. ```
  850. #### Docker
  851. Use Docker for spin up container for fetch data
  852. ```go
  853. type DockerConfig struct {
  854. Image string `yaml:"image" json:"image"`
  855. EntryPoint string `json:"entry_point" yaml:"entry_point"`
  856. Timeout uint32 `yaml:"timeout" json:"timeout"`
  857. Wait uint32 `yaml:"wait" json:"wait"`
  858. Flags []string `yaml:"flags" json:"flags"`
  859. Purge bool `json:"purge" yaml:"purge"`
  860. NoPull bool `yaml:"no_pull" json:"no_pull"`
  861. PullTimeout uint32 `yaml:"pull_timeout" json:"pull_timeout"`
  862. }
  863. ```
  864. **Docker default image**: docker.io/zenika/alpine-chrome
  865. - Image - image for the docker registry(provide with registry host)
  866. - EntryPoint - cmd which will be run inside container
  867. - Timeout[sec] - timeout for run container(without pulling image)
  868. - Wait[msec] - timeout of page loading (works just for Chromium based containers)
  869. - Flags - cmd arguments for run containers, default for Chromium based: "--no-sandbox","--headless", "--proxy-auto-detect", "--temp-profile", "--incognito", "--disable-logging", "--disable-gpu"
  870. - Purge - should we remove container after work done(like docker rm)
  871. - NoPull - prevent pulling of the image
  872. - PullTimeout - define timeout for pull contains
  873. ##### Environment variables
  874. 1. **DOCKER_HOST** - string - (EnvOverrideHost) to set the URL to the docker server.
  875. 2. **DOCKER_API_VERSION** - string - (EnvOverrideAPIVersion) to set the version of the API to use, leave empty for latest.
  876. 3. **DOCKER_CERT_PATH** - string - (EnvOverrideCertPath) to specify the directory from which to load the TLS certificates (ca.pem, cert.pem, key.pem).
  877. 4. **DOCKER_TLS_VERIFY** - bool - (EnvTLSVerify) to enable or disable TLS verification (off by default)
  878. Example:
  879. ```json
  880. {
  881. "wait": 10000,
  882. "image": "docker.io/zenika/alpine-chrome:with-node",
  883. "entry_point": "chromium-browser",
  884. "purge": true
  885. }
  886. ```
  887. #### Playwright
  888. Run browsers via playwright framework
  889. ```go
  890. type PlaywrightConfig struct {
  891. Browser PlaywrightBrowser `json:"browser" yaml:"browser"`
  892. Install bool `yaml:"install" json:"install"`
  893. Timeout uint32 `yaml:"timeout" json:"timeout"`
  894. Wait uint32 `yaml:"wait" json:"wait"`
  895. TypeOfWait *playwright.WaitUntilState `json:"type_of_wait" yaml:"type_of_wait"`
  896. PreRunScript string `json:"pre_run_script" yaml:"pre_run_script"`
  897. PostRunScript string `json:"post_run_script" yaml:"post_run_script"`
  898. Stealth bool `json:"stealth" yaml:"stealth"`
  899. StorageStateFile string `json:"storage_state_file" yaml:"storage_state_file"`
  900. IndexedDB bool `json:"indexed_db" yaml:"indexed_db"`
  901. Proxy *ProxyConfig `yaml:"proxy" json:"proxy"`
  902. }
  903. ```
  904. - Browser - enum["Chromium", "FireFox", "WebKit"] - which browser to use
  905. - Install - should we install browser (downloads the driver + browser matching the built-in `playwright-go` version on first use; not needed with the `ghcr.io/pxyup/fitter-mcp:playwright` image, which ships them preinstalled)
  906. - Timeout[sec] - timeout to run playwright
  907. - Wait[sec] - timeout of page loading
  908. - TypeOfWait - enum["load", "domcontentloaded", "networkidle", "commit"] which state of page we waiting, default is "load"
  909. - PreRunScript[""] - script which will be injected via AddInitScript and executed before any page script runs (on document creation, before navigation completes). Useful for patching the environment (navigator overrides, API stubs). Cannot access the loaded DOM. Also support placeholder [{PL}](#placeholder-list)
  910. - PostRunScript[""] - script which will be executed after page load, before reading content of the page. Useful for DOM interaction (clicks, scrolling). Also support placeholder [{PL}](#placeholder-list)
  911. - Stealth[false] - add script for trying passing bot defends
  912. - StorageStateFile[""] - path (supports `~/`) to a playwright storage state json (cookies + localStorage): loaded into the browser context before navigation, written back after every run so refreshed sessions stay alive. Lets headless runs reuse a real login — create the file once with [fitter_cli browser-login](#fitter_cli-browser-login--reuse-a-real-login-session). Use the same `browser` for login and scraping: sites may bind sessions to the browser fingerprint. Also support [formatting](#placeholder-list)
  913. - IndexedDB[false] - include IndexedDB in the persisted storage state (some SPAs, e.g. Firebase Auth, keep tokens there)
  914. - Proxy - setup proxy for request [config](#proxy-config)
  915. Example
  916. ```json
  917. {
  918. "timeout": 30,
  919. "wait": 30,
  920. "install": false,
  921. "browser": "Chromium"
  922. }
  923. ```
  924. ## Model
  925. With model we define result of the scrapping
  926. ```go
  927. type Model struct {
  928. ObjectConfig *ObjectConfig `yaml:"object_config" json:"object_config"`
  929. ArrayConfig *ArrayConfig `json:"array_config" yaml:"array_config"`
  930. BaseField *BaseField `json:"base_field" yaml:"base_field"`
  931. IsArray bool `json:"is_array" yaml:"is_array"`
  932. }
  933. ```
  934. Config can be one of:
  935. - [ObjectConfig](#objectconfig) - configuration of object format
  936. - [ArrayConfig](#arrayconfig) - configuration of array format
  937. - [BaseField](#basefield) - configuration of single/generated field
  938. - IsArray - bool[false] - force indicate that field is array(usable in case of [model field](#model-field) with [base field](#basefield))
  939. Example:
  940. ```json
  941. {
  942. "object_config": {}
  943. }
  944. ```
  945. ### ObjectConfig
  946. Configuration of the object and fields
  947. ```go
  948. type ObjectConfig struct {
  949. Fields map[string]*Field `json:"fields" yaml:"fields"`
  950. Field *BaseField `json:"field" yaml:"field"`
  951. ArrayConfig *ArrayConfig `json:"array_config" yaml:"array_config"`
  952. Condition string `json:"condition" yaml:"condition"`
  953. }
  954. ```
  955. - Condition - optional [condition](#conditional-fields) expression evaluated against the source node **before** resolution; when false the whole object is omitted from the parent (fields are not resolved at all)
  956. Config can be one of:
  957. - [Fields](#field) - map of each field definition; key - field name, value - configuration
  958. - [Field](#basefield) - used for element of array; fields which will be deserialized like basic type like "string", "int" and etc (used here for case array of basic types)
  959. - [ArrayConfig](#arrayconfig) - used for element of array; deserialization array of array
  960. Example:
  961. ```json
  962. {
  963. "fields": {
  964. "title": {
  965. "base_field": {
  966. "type": "string",
  967. "path": "type"
  968. }
  969. }
  970. }
  971. }
  972. ```
  973. ### ArrayConfig
  974. Configuration of the array and fields
  975. ```go
  976. type ArrayConfig struct {
  977. RootPath string `json:"root_path" yaml:"root_path"`
  978. Reverse bool `yaml:"reverse" json:"reverse"`
  979. ItemConfig *ObjectConfig `json:"item_config" yaml:"item_config"`
  980. LengthLimit uint32 `json:"length_limit" yaml:"length_limit"`
  981. Condition string `json:"condition" yaml:"condition"`
  982. ItemCondition string `json:"item_condition" yaml:"item_condition"`
  983. StaticConfig *StaticArrayConfig `json:"static_array" yaml:"static_array"`
  984. }
  985. ```
  986. - RootPath - selector for find root element of the array or repeated element in case of html parsing, size of array will be amount of children element under the root
  987. - Reverse - bool[false] - indicate that need use reverse iteration(n to 1)
  988. - LengthLimit - fixed size of the array (generated arrays only; not for static). Note: when the source has **fewer** elements than the limit, the array is **padded with trailing `null`s** to preserve the declared size (this is intentional) — omit `length_limit` to get exactly the source length instead
  989. - Condition - optional [condition](#conditional-fields) expression evaluated against the source node **before** resolution; when false the whole array is omitted from the parent
  990. - ItemCondition - optional [condition](#conditional-fields) expression evaluated against every **built** item (fRes - item value, fSrc - source element, fIndex - item index); items resolving to false are dropped from the array - declarative filtering. Not applied to static_array
  991. Config can be one of:
  992. - [ItemConfig](#objectconfig) - configuration of each element of the array
  993. - [StaticConfig](#static-array-config) - configuration of the static array
  994. Example:
  995. ```json
  996. {
  997. "root_path": "#content dt.quote > a",
  998. "item_config": {
  999. "field": {
  1000. "type": "string"
  1001. }
  1002. }
  1003. }
  1004. ```
  1005. #### Field
  1006. Common of the field
  1007. ```go
  1008. type Field struct {
  1009. BaseField *BaseField `json:"base_field" yaml:"base_field"`
  1010. ObjectConfig *ObjectConfig `json:"object_config" yaml:"object_config"`
  1011. ArrayConfig *ArrayConfig `json:"array_config" yaml:"array_config"`
  1012. FirstOf []*Field `json:"first_of" yaml:"first_of"`
  1013. }
  1014. ```
  1015. Config can be one of:
  1016. - [BaseField](#basefield) - fields which will be deserialized like basic type like "string", "int" and etc
  1017. - [ObjectConfig](#objectconfig) - in case our field in nested object
  1018. - [ArrayConfig](#arrayconfig) - in case our field in array
  1019. - [FirstOf](#field) - first not empty resolved field will be selected
  1020. Example:
  1021. ```json
  1022. {
  1023. "base_field": {
  1024. "type": "string",
  1025. "path": "div.current-temp span.heading"
  1026. }
  1027. }
  1028. ```
  1029. #### BaseField
  1030. In case we want get some static information or generate new one
  1031. ```go
  1032. type BaseField struct {
  1033. Type FieldType `yaml:"type" json:"type"`
  1034. Path string `yaml:"path" json:"path"`
  1035. HTMLAttribute string `json:"html_attribute" yaml:"html_attribute"`
  1036. Condition string `json:"condition" yaml:"condition"`
  1037. Generated *GeneratedFieldConfig `yaml:"generated" json:"generated"`
  1038. FirstOf []*BaseField `json:"first_of" yaml:"first_of"`
  1039. }
  1040. ```
  1041. - FieldType - enum["null", "boolean", "string", "int", "int64", "float", "float64", "array", "object", "html", "raw_string"] - static field for parse. **Important**: type html will only works from connector which return HTML (HTMLAttribute - have no effect in this case). [Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L25)
  1042. - Path - selector(relative in case it is array child) for parsing
  1043. - HTMLAttribute - extra value which have effect only in HTML parsing via **goquery**. Here you can specify which attribute need to be parsed.
  1044. - Condition - optional [condition](#conditional-fields) expression evaluated against the **extracted** value (fRes/fResJson/fResRaw, fIndex; fSrc - the node the field was resolved from, siblings included); when false the field is omitted from the parent object/array instead of producing null. Evaluated before [Generated](#generatedfieldconfig), so a false condition also skips generated work (sub-requests, file downloads)
  1045. **Important**: by default "string" type trimmed and all special chars is replaced, if you need plain string use "raw_string"
  1046. Config can be one of or empty:
  1047. - [Generated](#generatedfieldconfig) - field can be generated one which custom configuration
  1048. - [FirstOf](#basefield) - first not empty resolved field will be selected
  1049. Examples
  1050. ```json
  1051. {
  1052. "generated": {
  1053. "uuid": {}
  1054. }
  1055. }
  1056. ```
  1057. ```json
  1058. {
  1059. "type": "string",
  1060. "path": "text()"
  1061. }
  1062. ```
  1063. #### Conditional fields
  1064. Every field can carry a `condition` - an [expr-lang](https://expr-lang.org/) expression ([predefined values](#predefined-values)). When it evaluates to anything except `true` the field is **omitted** from the output (the key/item disappears), not set to `null`. An invalid expression also omits the field and logs an error.
  1065. Where the condition is evaluated:
  1066. - [BaseField](#basefield).`condition` - **after** extraction: `fRes` is the extracted value, `fSrc` the node the field was resolved from (its siblings included) - so `fSrc.on_sale == true` can gate a field on data you did not extract. A false condition skips [generated](#generatedfieldconfig) work entirely (no sub-request, no file download)
  1067. - [ObjectConfig](#objectconfig).`condition` / [ArrayConfig](#arrayconfig).`condition` - **before** resolution: `fRes`/`fSrc` are the source node (parsed value for json, text content for html)
  1068. - [ArrayConfig](#arrayconfig).`item_condition` - against every **built** item: `fRes` is the item, `fSrc` the source element it was built from, `fIndex` its index; false items are dropped - declarative array filtering. Use `fSrc` to filter on source attributes without adding them to the output
  1069. Filter array items - `fSrc.in_stock` reads the source element (not extracted into the output), `fRes.price` the built item:
  1070. ```json
  1071. {
  1072. "array_config": {
  1073. "root_path": "products",
  1074. "item_condition": "fSrc.in_stock && fRes.price > 0",
  1075. "item_config": {
  1076. "fields": {
  1077. "title": { "base_field": { "type": "string", "path": "title" } },
  1078. "price": { "base_field": { "type": "float", "path": "price" } }
  1079. }
  1080. }
  1081. }
  1082. }
  1083. ```
  1084. Omit a key unless the value passes a check:
  1085. ```json
  1086. {
  1087. "discount": {
  1088. "base_field": {
  1089. "type": "float",
  1090. "path": "discount_pct",
  1091. "condition": "fRes > 0"
  1092. }
  1093. }
  1094. }
  1095. ```
  1096. Special cases:
  1097. - in a [static array](#static-array-config) an omitted item stays `null` (positions are fixed by definition, indexes never shift)
  1098. - if the **root** model config is omitted the parse result is `null`
  1099. - inside [first_of](#basefield) a branch with a false condition counts as empty, so the next branch is tried
  1100. Runnable example: [examples/config_conditions.json](https://github.com/PxyUp/fitter/blob/master/examples/config_conditions.json)
  1101. #### GeneratedFieldConfig
  1102. Provide functionality of generating field on the flight
  1103. ```go
  1104. type GeneratedFieldConfig struct {
  1105. UUID *UUIDGeneratedFieldConfig `yaml:"uuid" json:"uuid"`
  1106. Static *StaticGeneratedFieldConfig `yaml:"static" json:"static"`
  1107. Formatted *FormattedFieldConfig `json:"formatted" yaml:"formatted"`
  1108. Plugin *PluginFieldConfig `yaml:"plugin" json:"plugin"`
  1109. Calculated *CalculatedConfig `yaml:"calculated" json:"calculated"`
  1110. File *FileFieldConfig `yaml:"file" json:"file"`
  1111. Model *ModelField `yaml:"model" json:"model"`
  1112. FileStorageField *FileStorageField `json:"file_storage" yaml:"file_storage"`
  1113. }
  1114. ```
  1115. Config can be one of:
  1116. - [UUID](#uuid) - generate random UUID V4
  1117. - [Static](#static) - generate static field
  1118. - [Formatted](#formatted-field-config) - format field
  1119. - [Model](#model-field) - model generated from the other connector and model
  1120. - [Plugin](#plugin-field) - plugin field
  1121. - [Calculated](#calculated-field) - calculated field
  1122. - [File](#file-field) - file field (for download file from server)
  1123. - [FileStorage](#file-storage-field) - file field which can be saved to local file
  1124. Examples:
  1125. ```json
  1126. {
  1127. "uuid": {}
  1128. }
  1129. ```
  1130. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L58
  1131. ```json
  1132. {
  1133. "model": {
  1134. "type": "array",
  1135. "model": {
  1136. "array_config": {
  1137. "root_path": "#content dt.quote > a",
  1138. "item_config": {
  1139. "field": {
  1140. "type": "string"
  1141. }
  1142. }
  1143. }
  1144. },
  1145. "connector_config": {
  1146. "response_type": "HTML",
  1147. "url": "http://www.quotationspage.com/random.php",
  1148. "attempts": 3,
  1149. "browser_config": {
  1150. "chromium": {
  1151. "path": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
  1152. "wait": 10000
  1153. }
  1154. }
  1155. }
  1156. }
  1157. }
  1158. ```
  1159. #### UUID
  1160. Generate random UUID V4 on the flight, can be used for generate uniq id
  1161. ```go
  1162. type UUIDGeneratedFieldConfig struct {
  1163. Regexp string `yaml:"regexp" json:"regexp"`
  1164. }
  1165. ```
  1166. - Regexp - provide matcher which can be used for get part of generated uuid
  1167. #### Static
  1168. Generate static field
  1169. ```go
  1170. type StaticGeneratedFieldConfig struct {
  1171. Type FieldType `yaml:"type" json:"type"`
  1172. Value string `json:"value" yaml:"value"`
  1173. Raw json.RawMessage `json:"raw" yaml:"raw"`
  1174. }
  1175. ```
  1176. - Type - enum["null", "boolean", "string", "int","int64","float","float64", "array", "object"] - type of the field
  1177. - Value - string value of the field
  1178. - Raw - pure json value of the field
  1179. Example
  1180. ```json
  1181. {
  1182. "type": "int",
  1183. "value": "65"
  1184. }
  1185. ```
  1186. ```json
  1187. {
  1188. "type": "array",
  1189. "value": "[65,45]"
  1190. }
  1191. ```
  1192. ```json
  1193. {
  1194. "type": "array",
  1195. "raw": [65,45]
  1196. }
  1197. ```
  1198. #### Formatted Field Config
  1199. Generate formatted field which will pass value from parent [base field](#basefield)
  1200. ```go
  1201. type FormattedFieldConfig struct {
  1202. Template string `yaml:"template" json:"template"`
  1203. }
  1204. ```
  1205. - Template - template in with placeholder [{PL}](#placeholder-list) where [parent](#basefield) value will be injected like string
  1206. Example:
  1207. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L98
  1208. ```json
  1209. {
  1210. "template": "https://news.ycombinator.com/item?id={PL}"
  1211. }
  1212. ```
  1213. #### File Storage Field
  1214. Field can be used for store field result as local file
  1215. ```go
  1216. type FileStorageField struct {
  1217. Content string `json:"content" yaml:"content"`
  1218. Raw json.RawMessage `yaml:"raw" yaml:"raw"`
  1219. FileName string `json:"file_name" yaml:"file_name"`
  1220. Path string `json:"path" yaml:"path"`
  1221. Append bool `json:"append" yaml:"append"`
  1222. }
  1223. ```
  1224. - Content - template string for content. Important: can be with [inject of the parent value as a string](#placeholder-list)
  1225. - Raw - raw json content of field. Important: can be with [inject of the parent value as a string](#placeholder-list)
  1226. - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with [inject of the parent value as a string](#placeholder-list).
  1227. - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with [inject of the parent value as a string](#placeholder-list)
  1228. - Append[false] - append to file or not
  1229. ```json
  1230. {
  1231. "content": "{{{id}}}, {{{message}}}\n",
  1232. "append": true,
  1233. "file_name": "{{{id}}}.csv",
  1234. "path": "/Users/pxyup/fitter/examples/cli/test/csv"
  1235. }
  1236. ```
  1237. #### File Field
  1238. Field can be used for download file from server locally
  1239. ```go
  1240. type FileFieldConfig struct {
  1241. Config *ServerConnectorConfig `yaml:"config" json:"config"`
  1242. Url string `yaml:"url" json:"url"`
  1243. FileName string `json:"file_name" yaml:"file_name"`
  1244. Path string `json:"path" yaml:"path"`
  1245. }
  1246. ```
  1247. - Config - [ServerConfig](#serverconnectorconfig) use default fitter http.Client for send request
  1248. - Url - url of the image. Important: URL in the connector can be with [inject of the parent value as a string](#placeholder-list)
  1249. - FileName - local file name for storing file. By default, it is try get FileName from header, after that from url. Important: can be with [inject of the parent value as a string](#placeholder-list).
  1250. - Path - local file parent directory for storing file. Default path it is process directory. Important: can be with [inject of the parent value as a string](#placeholder-list)
  1251. Result of the field will be local file path as string
  1252. ```json
  1253. {
  1254. "url": "https://images.shcdn.de/resized/w680/p/dekostoff-gobelinstoff-panel-oriental-cat-46-x-46_P19-KP_2.jpg",
  1255. "path": "/Users/pxyup/fitter/bin",
  1256. "config": {
  1257. "method": "GET"
  1258. }
  1259. }
  1260. ```
  1261. With propagated URL ([inject of the parent value as a string](#placeholder-list))
  1262. ```json
  1263. {
  1264. "url": "https://picsum.photos{PL}",
  1265. "path": "/Users/pxyup/fitter/bin",
  1266. "config": {
  1267. "method": "GET"
  1268. }
  1269. }
  1270. ```
  1271. Config example:
  1272. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image.json
  1273. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_image_multiple.json
  1274. #### Calculated field
  1275. Field can generate different types depends from expression
  1276. ```go
  1277. type CalculatedConfig struct {
  1278. Type FieldType `yaml:"type" json:"type"`
  1279. Expression string `yaml:"expression" json:"expression"`
  1280. }
  1281. ```
  1282. - Type - resulting type of expression\
  1283. - Expression - expression for calculation (we use [this lib](https://github.com/expr-lang/expr) for calculated expression)
  1284. ##### Predefined values
  1285. **FNull** - alias for builder.Nullvalue
  1286. **FNil** - alias for nil
  1287. **isNull(value T)** - function for check is value is FNull
  1288. **fRes** - it is raw(with proper type) result from the parsing [base field](#basefield)
  1289. **fIndex** - it is index in parent array(only if parent was array field)
  1290. **fResJson** - it is JSON string representation of the raw result
  1291. **fResRaw** - result in bytes format
  1292. **fSrc** - only in [condition/item_condition](#conditional-fields) expressions: the source node the value was resolved from (parsed value for json - siblings included, text content for html). Not available in calculated/formatted/notifier expressions
  1293. **FNewLine** - new line separator
  1294. ```json
  1295. {
  1296. "type": "bool",
  1297. "expression": "fRes > 500"
  1298. }
  1299. ```
  1300. #### Plugin field
  1301. Field can be some external plugin for fitter
  1302. [More](https://github.com/PxyUp/fitter/blob/master/examples/plugin/README.md)
  1303. ```go
  1304. type PluginFieldConfig struct {
  1305. Name string `json:"name" yaml:"name"`
  1306. Config json.RawMessage `json:"config" yaml:"config"`
  1307. }
  1308. ```
  1309. - Name - name of the plugin(without extension just name)
  1310. - Config - json config of the plugin
  1311. #### Model Field
  1312. Field type which can be generated on the flight by news [model](#model) and [connector](#connector)
  1313. ```go
  1314. type ModelField struct {
  1315. // Type of parsing
  1316. ConnectorConfig *ConnectorConfig `yaml:"connector_config" json:"connector_config"`
  1317. // Model of the response
  1318. Model *Model `yaml:"model" json:"model"`
  1319. Type FieldType `yaml:"type" json:"type"`
  1320. Path string `yaml:"path" json:"path"`
  1321. Expression string `yaml:"expression" json:"expression"`
  1322. }
  1323. ```
  1324. - [ConnectorConfig](#connector) - which connector to use. Important: URL in the connector can be with [inject of the parent value as a string](#placeholder-list)
  1325. - [Model](#model) - configuration of the underhood model
  1326. - Type - enum["null", "boolean", "string", "int", "int64", "float", "float64", "array", "object"] - type of generated field
  1327. - Path - in case we cant extract some information from generated field we can use json selector for extract
  1328. - Expression - string which can be used for post processing of the Model (**ignoring path field**)
  1329. Examples:
  1330. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_cli.json#L60
  1331. ```json
  1332. {
  1333. "type": "array",
  1334. "model": {
  1335. "array_config": {
  1336. "root_path": "#content dt.quote > a",
  1337. "item_config": {
  1338. "field": {
  1339. "type": "string"
  1340. }
  1341. }
  1342. }
  1343. }
  1344. }
  1345. ```
  1346. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_weather.json#L37
  1347. ```json
  1348. {
  1349. "type": "string",
  1350. "path": "temp.temp",
  1351. "model": {
  1352. "object_config": {
  1353. "fields": {
  1354. "temp": {
  1355. "base_field": {
  1356. "type": "string",
  1357. "path": "//div[@id='forecast_list_ul']//td/b/a/@href",
  1358. "generated": {
  1359. "model": {
  1360. "type": "string",
  1361. "model": {
  1362. "object_config": {
  1363. "fields": {
  1364. "temp": {
  1365. "base_field": {
  1366. "type": "string",
  1367. "path": "div.current-temp span.heading"
  1368. }
  1369. }
  1370. }
  1371. }
  1372. },
  1373. "connector_config": {
  1374. "response_type": "HTML",
  1375. "attempts": 4,
  1376. "url": "https://openweathermap.org{PL}",
  1377. "browser_config": {
  1378. "playwright": {
  1379. "timeout": 30,
  1380. "wait": 30,
  1381. "install": false,
  1382. "browser": "FireFox",
  1383. "type_of_wait": "networkidle"
  1384. }
  1385. }
  1386. }
  1387. }
  1388. }
  1389. }
  1390. }
  1391. }
  1392. }
  1393. },
  1394. "connector_config": {
  1395. "response_type": "xpath",
  1396. "attempts": 3,
  1397. "url": "https://openweathermap.org/find?q={PL}",
  1398. "browser_config": {
  1399. "playwright": {
  1400. "timeout": 30,
  1401. "wait": 30,
  1402. "install": false,
  1403. "browser": "Chromium"
  1404. }
  1405. }
  1406. }
  1407. }
  1408. ```
  1409. #### Static Array Config
  1410. Provide static(fixed length) array generation
  1411. ```go
  1412. type StaticArrayConfig struct {
  1413. Items map[uint32]*Field `yaml:"items" json:"items"`
  1414. Length uint32 `yaml:"length" json:"length"`
  1415. }
  1416. ```
  1417. - [Items](#field) - map[uint32]*[Field](#field) - key is index in array, value is field definition
  1418. - Length - if set(1+) can be used for define custom length of array
  1419. Examples:
  1420. ```json
  1421. {
  1422. "0": {
  1423. "base_field": {
  1424. "type": "string",
  1425. "path": "div.current-temp span.heading"
  1426. }
  1427. }
  1428. }
  1429. ```
  1430. ```json
  1431. {
  1432. "length": 4,
  1433. "0": {
  1434. "base_field": {
  1435. "type": "string",
  1436. "path": "div.current-temp span.heading"
  1437. }
  1438. }
  1439. }
  1440. ```
  1441. ```json
  1442. {
  1443. "length": 4,
  1444. "2": {
  1445. "base_field": {
  1446. "type": "string",
  1447. "path": "div.current-temp span.heading"
  1448. }
  1449. }
  1450. }
  1451. ```
  1452. #### Placeholder list
  1453. 1. {PL} - for inject value
  1454. 2. {INDEX} - for inject index in parent array
  1455. 3. {HUMAN_INDEX} - for inject index in parent array in human way
  1456. 4. {{{json_path}}} - will get information from propagated "object"/"array" field
  1457. 5. {{{RefName=SomeName}}} - get [reference](#references) value by name. [Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L67)
  1458. 6. {{{RefName=SomeName json.path}}} - get [reference](#references) value by name and extract value by json path. [Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L67)
  1459. 7. {{{FromEnv=ENV_KEY}}} - get value from environment variable
  1460. 8. {{{FromExp=fRes + 5 + fIndex}}} - get value from the [expression](https://github.com/expr-lang/expr). [Predefined values](#predefined-values)
  1461. 9. {{{FromInput=.}}} or {{{FromInput=json.path}}} - get value from input of trigger or library
  1462. 10. {{{FromFile=./test_file.log}}} - get value from file by path. Content of file also can contain placeholders
  1463. 11. {{{FromURL=http://localhost:8081}}} - get response from url
  1464. Examples:
  1465. ```text
  1466. {{{FromExp="{{{FromEnv=TEST_VAL}}}" + "hello"}}}
  1467. ```
  1468. ```text
  1469. Current time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}}
  1470. ```
  1471. ```text
  1472. Current time is: {PL} with token from TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}}
  1473. ```
  1474. ```text
  1475. TokenRef={{{RefName=TokenRef}}} and TokenObjectRef={{{RefName=TokenObjectRef token}}} Object={{{value}}} {PL} Env={{{FromEnv=TEST_VAL}}} {INDEX} {HUMAN_INDEX}
  1476. ```
  1477. ## References
  1478. Special map which **prefetched**(before any processing) and can be user for [connector](#referenceconnectorconfig) or for [placeholder](#placeholder-list)
  1479. Can be used for:
  1480. 1. Cache jwt token and use them in headers
  1481. 2. Cache values
  1482. 3. Etc
  1483. ### Reference
  1484. ```go
  1485. type Reference struct {
  1486. *ModelField
  1487. Expire *uint32 `yaml:"expire" json:"expire"`
  1488. }
  1489. ```
  1490. - [ModelField](#model-field) - is embedded struct, you can use same fields
  1491. - Expire[sec] - duration when reference is expired after fetching. Not set => **forever cached**. Set to 0 => **every time re-fetch**. Set to n > 0 => **cached for n second**
  1492. For [Fitter](#how-to-use-fitter)
  1493. ```go
  1494. type RefMap map[string]*Reference
  1495. type Config struct {
  1496. // Other Config Fields
  1497. Limits *Limits `yaml:"limits" json:"limits"`
  1498. References RefMap `json:"references" yaml:"references"`
  1499. }
  1500. ```
  1501. For [Fitter Cli](#how-to-use-fittercli)
  1502. ```go
  1503. type RefMap map[string]*Reference
  1504. type CliItem struct {
  1505. // Other Config Fields
  1506. Limits *Limits `yaml:"limits" json:"limits"`
  1507. References RefMap `json:"references" yaml:"references"`
  1508. }
  1509. ```
  1510. - References - map[string]*Reference - object where is key if ReferenceName (can be user for [connector](#referenceconnectorconfig) or [placeholder](#placeholder-list)) and value is [Reference](#reference)
  1511. - [Limits](#limits)
  1512. Example
  1513. https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json#L2
  1514. ```json
  1515. {
  1516. "references": {
  1517. "TokenRef": {
  1518. "expire": 10,
  1519. "connector_config": {
  1520. "response_type": "json",
  1521. "static_config": {
  1522. "value": "\"plain token\""
  1523. }
  1524. },
  1525. "model": {
  1526. "base_field": {
  1527. "type": "string"
  1528. }
  1529. }
  1530. },
  1531. "TokenObjectRef": {
  1532. "connector_config": {
  1533. "response_type": "json",
  1534. "static_config": {
  1535. "value": "{\"token\":\"token from object\"}"
  1536. }
  1537. },
  1538. "model": {
  1539. "object_config": {
  1540. "fields": {
  1541. "token": {
  1542. "base_field": {
  1543. "type": "string",
  1544. "path": "token"
  1545. }
  1546. }
  1547. }
  1548. }
  1549. }
  1550. }
  1551. }
  1552. }
  1553. ```
  1554. [Example](https://github.com/PxyUp/fitter/blob/master/examples/cli/config_ref.json)
  1555. ## Notifiers
  1556. Optional per-item config `item.notifier_config` which pushes the parse result somewhere after processing. The result is still returned as usual (CLI/MCP output, service logs); the notifier additionally delivers it. Works in Fitter (service mode), Fitter_CLI and Fitter_MCP.
  1557. ```go
  1558. type NotifierConfig struct {
  1559. Expression string `yaml:"expression" json:"expression"`
  1560. Force bool `json:"force" yaml:"force"`
  1561. SendArrayByItem bool `yaml:"send_array_by_item" json:"send_array_by_item"`
  1562. Template string `yaml:"template" json:"template"`
  1563. // exactly ONE destination:
  1564. Console *ConsoleConfig `yaml:"console" json:"console"`
  1565. TelegramBot *TelegramBotConfig `yaml:"telegram_bot" json:"telegram_bot"`
  1566. Http *HttpConfig `yaml:"http" json:"http"`
  1567. Redis *RedisNotifierConfig `json:"redis" yaml:"redis"`
  1568. File *FileStorageField `json:"file" yaml:"file"`
  1569. }
  1570. ```
  1571. - Expression - optional [expr-lang](https://expr-lang.org/) condition: notify only when it evaluates to true. The parse result is available as `fRes` (parsed value), `fResRaw` (raw bytes), `fResJson` (JSON string), e.g. `len(fResRaw) > 0`
  1572. - Force - notify even if parsing finished with an error
  1573. - SendArrayByItem - if the result is an array, send each element as a separate notification
  1574. - Template - optional template applied to the result before sending, [placeholders](#placeholder-list) allowed
  1575. - Destination - exactly one of `console`, `telegram_bot`, `http`, `redis`, `file`
  1576. Destination configs:
  1577. ```go
  1578. type HttpConfig struct {
  1579. Url string `yaml:"url" json:"url"`
  1580. Method string `json:"method" yaml:"method"`
  1581. Headers map[string]string `yaml:"headers" json:"headers"`
  1582. Timeout uint32 `yaml:"timeout" json:"timeout"`
  1583. }
  1584. type TelegramBotConfig struct {
  1585. Token string `json:"token" yaml:"token"`
  1586. UsersId []int64 `json:"users_id" yaml:"users_id"`
  1587. Pretty bool `json:"pretty" yaml:"pretty"`
  1588. OnlyMsg bool `json:"only_msg" yaml:"only_msg"`
  1589. }
  1590. type RedisNotifierConfig struct {
  1591. Addr string `json:"addr" yaml:"addr"`
  1592. Password string `json:"password" yaml:"password"`
  1593. DB int `json:"db" yaml:"db"`
  1594. Channel string `json:"channel" yaml:"channel"`
  1595. }
  1596. type ConsoleConfig struct {
  1597. OnlyResult bool `json:"only_result" yaml:"only_result"`
  1598. }
  1599. ```
  1600. The `file` destination uses the same [FileStorageField](#file-storage-field) as the file field type.
  1601. Example ([examples/config_telegram.json](https://github.com/PxyUp/fitter/blob/master/examples/config_telegram.json)):
  1602. ```json
  1603. {
  1604. "item": {
  1605. "connector_config": { "...": "..." },
  1606. "model": { "...": "..." },
  1607. "notifier_config": {
  1608. "expression": "len(fResRaw) > 0",
  1609. "telegram_bot": {
  1610. "token": "{{{FromEnv=TG_TOKEN}}}",
  1611. "users_id": [123456],
  1612. "pretty": true
  1613. }
  1614. }
  1615. }
  1616. }
  1617. ```
  1618. ## Limits
  1619. Provide limitation for prevent DDOS, big usage of memory
  1620. ```go
  1621. type Limits struct {
  1622. HostRequestLimiter HostRequestLimiter `yaml:"host_request_limiter" json:"host_request_limiter"`

Lines 2,001–2,024 are not shown. The link to the file on GitHub has every line.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.