wigolo
Monitor a page for changes over time. Create lazy watch jobs on one or many URLs, list them, and check on demand with optional SSRF-guarded webhook delivery.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Check it deeper on your machine
It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.
As a Claude Code plugin, install it and then ask Claude Code:
/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/KnockOutEZ/wigolo with scanskillsafety
Or from a terminal, clone it and run it:
git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/KnockOutEZ/wigolo'
It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.
Badge for your README
Markdown
[](/listings/knockoutez--wigolo)HTML
<a href="/listings/knockoutez--wigolo"><img src="/api/listings/knockoutez--wigolo/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
npx wigolo init # set up the local engine — any system npx wigolo init --agents=claude-code,cursor # …or set up + wire your day-to-day agents in one command
Requires **Node ≥ 20** and ~1.5 GB of free disk on macOS, Linux, or Windows. Bare `init` sets up the local engine: it downloads the browser engine and on-device models, runs a health check, and reports each component. Adding `--agents` wires the named agents in the same run, so a coding agent you use daily is ready in one command.
- **Supported agents** — `--agents` takes any of `claude-code` · `cursor` · `codex` · `gemini-cli` · `opencode` · `vscode` · `windsurf` · `zed` · `antigravity` (comma-separated); wigolo writes the MCP config and, where supported, instructions for each. - **Any other setup** — any MCP client, agent framework, or self-hosted agent registers `npx -y wigolo` in its own MCP config. The [installation guide](docs/installation.md) has the exact config block for every client, plus Docker, Homebrew, and single-file-binary channels. - **More on the way** — the supported list keeps growing, and a PR to add your agent is welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). - **Interactive setup** — `--interactive` is a plain-text flow; `--wizard` is the full terminal TUI. - **Defer downloads** — `--no-warmup` waits until first use. A failed component download never fails setup; init reports what's not ready with the exact fix and still completes.
`init` is unattended by default, so it's safe in scripts and CI, and any setup problem surfaces right here in the per-component report, before your agent's first call. **Search, fetch, crawl, extract, cache, and find-similar work with no API key.** Check it's healthy anytime:
npx wigolo doctor
To remove everything cleanly, run `npx wigolo config --uninstall --yes`. You can also paste the [installation guide](docs/installation.md) into any AI assistant and let it do the setup; it's written to be self-contained.
Recommended — a free key for `research` & `agent`
Search, fetch, crawl, extract, cache, and find-similar are **fully keyless**. `research`, `agent`, and `search format=answer` use an LLM to write the synthesized, cited answer. Without one they hand back a raw brief and evidence for your agent to assemble. A free Gemini key turns that into a finished answer:
export WIGOLO_LLM_PROVIDER=gemini export GEMINI_API_KEY=<free-key> # grab one at aistudio.google.com/apikey — the free tier is plenty
Any provider works (`anthropic` · `openai` · `groq`), or stay fully local and keyless with `WIGOLO_LLM_PROVIDER=ollama` (or any OpenAI-compatible URL). Set it in your shell or your agent's MCP `env` block. Providers, models, and the keyless local-model ladder are in the [configuration guide](docs/configuration.md).
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 424 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- ---
- name: wigolo
- description: Local-first web intelligence MCP server for AI coding agents. Ten tools for search, fetch, crawl, cache, extract, find similar, research, agent-driven data gathering, page diffing, and change watching. No API keys. Results cached in a local knowledge store.
- author: KnockOutEZ
- license: AGPL-3.0-only
- repository: https://github.com/KnockOutEZ/wigolo
- transport: stdio
- install: npx wigolo
- runtime: node
- min_runtime_version: "20"
- tools:
- - name: fetch
- description: Fetch one URL, return clean markdown. Auto-routes between HTTP and browser engine. Supports sections, auth, screenshots, browser actions.
- - name: search
- description: Search the web, return extracted markdown per result. Single query or array of query variants. Domain, category, date filters. Formats include ML-scored highlights with citations for host-LLM synthesis.
- - name: crawl
- description: Crawl a site from a seed URL. BFS, DFS, sitemap, or map (URL-only) strategies with regex include/exclude filters.
- - name: cache
- description: Full-text search over previously fetched content. URL glob, date filters, stats, clear, and change detection via re-fetch.
- - name: extract
- description: Structured extraction from URL or raw HTML. Modes: selector (CSS), tables, metadata (meta + JSON-LD), schema (heuristic field matching), structured (tables + dl + JSON-LD + chart hints + key-value pairs in one call).
- - name: find_similar
- description: Find pages similar to a URL or concept. Hybrid cache (keyword search + embeddings) + optional web supplement.
- - name: research
- description: Multi-step research pipeline. Question decomposition, parallel sub-search, source synthesis with citations. Quick, standard, or comprehensive depth.
- - name: agent
- description: Natural-language data gathering. Plans searches/URLs, fetches in parallel within page and time budgets, optionally applies a JSON Schema to each page.
- - name: diff
- description: Compare two page versions — a live URL vs its cached copy, two URLs, or two markdown blobs. Unified patch, per-section hunks, or a counts summary at line, word, or section granularity.
- - name: watch
- description: Monitor a page for changes over time. Create lazy watch jobs on one or many URLs, list them, and check on demand with optional SSRF-guarded webhook delivery.
- ---
- # wigolo
- Local-first web intelligence MCP server for AI coding agents. Ships ten tools over stdio. All network results land in a local knowledge cache.
- ## Host-LLM synthesis (read me first)
- Wigolo has no internal LLM. It returns *structured evidence* so the calling model (you) writes the final answer. Fold structure into your reply rather than collapsing it away:
- - `search` with `format: "highlights"` — ML-scored passages + `citations`. Quote and cite [N].
- - `research` — when MCP sampling is unavailable (common), the output carries a `brief` with `topics`, `highlights`, `key_findings`. Use it as the scaffold for the report you write.
- - `find_similar` — may return a `cold_start` string. Pass it to the user; it explains why results came from the web and how to warm the cache.
- - `extract` with `mode: "structured"` — one call for tables + `<dl>` definitions + JSON-LD + chart hints + key-value pairs.
- - `fetch` metadata — surfaces `og_type`, `canonical_url`, and `og_image`; use `canonical_url` to dedupe tracked/canonical URLs.
- ## Quick Setup
- **Claude Code:**
- ```bash
- claude mcp add wigolo -- npx wigolo
- ```
- **Cursor / VS Code / any MCP client:**
- ```json
- {
- "mcpServers": {
- "wigolo": {
- "command": "npx",
- "args": ["wigolo"]
- }
- }
- }
- ```
- **Warmup (recommended, one-time):**
- ```bash
- npx wigolo warmup # installs browser engine + bootstraps search engine
- npx wigolo warmup --all # also installs Firefox, WebKit, ML reranker, and embeddings
- npx wigolo warmup --force # wipe search engine state and rebuild
- ```
- Warmup flags: `--force`, `--all`, `--reranker`, `--firefox`, `--webkit`, `--embeddings`, `--no-searxng`, `--verify`.
- ## Tools
- ### fetch
- Fetch a single URL and return clean markdown. Use when you already have a specific URL.
- Parameters:
- - `url` (string, required)
- - `render_js`: `"auto"` (default) | `"always"` | `"never"`
- - `use_auth`: boolean (default `false`) — reuses the user's browser session
- - `max_chars`: number
- - `section`: string — return only the content under a heading
- - `section_index`: number (default `0`) — which heading match when multiple hit
- - `screenshot`: boolean (default `false`)
- - `headers`: object
- - `force_refresh`: boolean — bypass cache
- - `actions`: array of `{type, selector, text, ms, timeout, direction, amount}` — `click`, `type`, `wait`, `wait_for`, `scroll`, `screenshot`. Forces browser rendering when present.
- Example:
- ```json
- { "url": "https://react.dev/reference/react/useState", "section": "Parameters" }
- ```
- Tip: `section` is much cheaper than reading the full page. Repeat fetches of the same URL are free from cache unless `force_refresh: true`.
- ### search
- Search the web and return extracted markdown per result. Use when you don't have a URL yet.
- Parameters:
- - `query` (string OR `string[]`, required) — array runs variants in parallel and dedupes
- - `max_results`: number (default `5`, cap `20`)
- - `include_content`: boolean (default `true`)
- - `content_max_chars`: number (default `30000`)
- - `max_total_chars`: number (default `50000`)
- - `time_range`: `"day"` | `"week"` | `"month"` | `"year"`
- - `include_domains` / `exclude_domains`: `string[]`
- - `from_date` / `to_date`: ISO `YYYY-MM-DD`
- - `category`: `"general"` | `"news"` | `"code"` | `"docs"` | `"papers"` | `"images"`
- - `language`: string
- - `search_engines`: `string[]` — override engine selection
- - `format`: `"full"` (default) | `"context"` (token-budgeted string) | `"highlights"` (ML-scored passages + citations) | `"answer"` (synthesized via MCP sampling; falls back to `highlights` when unsupported) | `"stream_answer"` (answer + phase progress notifications)
- - `max_highlights`: number (default `10`) — cap when `format: "highlights"`
- - `force_refresh`: boolean
- Example:
- ```json
- { "query": ["react server components patterns", "RSC data fetching", "react server components streaming"], "category": "docs", "include_domains": ["react.dev"], "max_results": 5 }
- ```
- Tip: keyword queries beat natural-language questions. A 3–5 item `query` array usually finds more unique sources than one longer query.
- ### crawl
- Crawl a site starting from a seed URL.
- Parameters:
- - `url` (string, required)
- - `strategy`: `"bfs"` (default) | `"dfs"` | `"sitemap"` | `"map"` (URL-only discovery, no content)
- - `max_depth`: number (default `2`)
- - `max_pages`: number (default `20`)
- - `include_patterns` / `exclude_patterns`: regex `string[]`
- - `use_auth`: boolean (default `false`)
- - `extract_links`: boolean (default `false`) — returns inter-page link graph
- - `max_total_chars`: number (default `100000`)
- Example:
- ```json
- { "url": "https://docs.python.org/3/library/", "strategy": "sitemap", "max_pages": 30, "include_patterns": ["^https://docs\\.python\\.org/3/library/asyncio"] }
- ```
- Tip: `strategy: "sitemap"` is faster and more complete than BFS on doc sites. `strategy: "map"` returns URLs only — cheap way to scope before targeted fetches.
- ### cache
- Search previously fetched content without hitting the network.
- Parameters:
- - `query`: full-text search — supports `AND`, `OR`, `NOT`, `"exact phrase"`
- - `url_pattern`: glob (e.g. `"*react.dev*"`)
- - `since`: ISO date
- - `stats`: boolean — returns total URLs, size, date range
- - `clear`: boolean — deletes matching entries (requires one of `query`, `url_pattern`, `since`)
- - `check_changes`: boolean — re-fetches matching URLs, reports changed/unchanged with diff summaries
- Example:
- ```json
- { "query": "useState OR useReducer", "url_pattern": "*react.dev*" }
- ```
- Tip: cache hits are instant and cross-session. Run this before `search` or `fetch` when you suspect the content is already on disk.
- ### extract
- Structured extraction from URL or raw HTML.
- Parameters:
- - `url` OR `html` (one required; `url` wins if both provided)
- - `mode`: `"metadata"` (default) | `"selector"` | `"tables"` | `"schema"` | `"structured"` (tables + `<dl>` definitions + JSON-LD + chart hints + microdata/data-attr/grid key-value pairs in one call)
- - `css_selector`: string — required for `mode: "selector"`
- - `multiple`: boolean (default `false`) — return all matches, selector mode only
- - `schema`: JSON Schema object with `properties` — required for `mode: "schema"`
- Example:
- ```json
- { "url": "https://example.com/product", "mode": "schema", "schema": { "type": "object", "properties": { "price": { "type": "string" }, "name": { "type": "string" }, "sku": { "type": "string" } } } }
- ```
- Tip: `mode: "schema"` does heuristic matching over CSS classes, ARIA labels, microdata, and JSON-LD — no LLM call required. `mode: "structured"` returns every structured pattern on the page (`tables`, `definitions`, `jsonld`, `chart_hints`, `key_value_pairs`) in one response — prefer it over chaining multiple extract calls.
- ### find_similar
- Find pages related to a URL or a free-text concept.
- Parameters:
- - `url` OR `concept` (one required)
- - `max_results`: number (default `10`, cap `50`)
- - `include_domains` / `exclude_domains`: `string[]`
- - `include_cache`: boolean (default `true`)
- - `include_web`: boolean (default `true`)
- Example:
- ```json
- { "url": "https://react.dev/reference/react/useState", "max_results": 8, "include_domains": ["react.dev", "developer.mozilla.org"] }
- ```
- Tip: uses hybrid 3-way RRF fusion — keyword search + semantic embeddings + live web search. Each result carries `match_signals` with `embedding_rank`, `fts5_rank`, and `fused_score`. If the cache is empty or embeddings aren't set up, the response includes a `cold_start` string — pass it to the user to explain why results came from the web.
- ### research
- Multi-step research pipeline with decomposition, parallel search, and cited synthesis.
- Parameters:
- - `question` (string, required)
- - `depth`: `"quick"` (~15s, 2 sub-queries, 5–8 sources) | `"standard"` (~40s, default) | `"comprehensive"` (~80s, 7 sub-queries, 20–25 sources)
- - `max_sources`: number (cap `50`) — overrides depth default
- - `include_domains` / `exclude_domains`: `string[]`
- - `schema`: JSON Schema — if present, report is structured to fill these fields
- - `stream`: boolean — emit progress notifications per phase
- Example:
- ```json
- { "question": "How do modern JS bundlers tree-shake ESM vs CJS?", "depth": "standard", "include_domains": ["webpack.js.org", "rollupjs.org", "esbuild.github.io", "vitejs.dev"] }
- ```
- Tip: `research` checks cache internally — no need to pre-probe. With MCP sampling, the tool synthesizes the report directly. Without sampling (the common case), the output ships a `brief` with `topics`, `highlights`, and `key_findings`, plus the raw sources — the host LLM writes the final report from the brief.
- ### agent
- Natural-language data gathering. Plans queries and URLs from a prompt, runs them in parallel within budget, optionally applies a schema.
- Parameters:
- - `prompt` (string, required)
- - `urls`: `string[]` — seed URLs to include
- - `schema`: JSON Schema — extract structured fields per page and merge
- - `max_pages`: number (default `10`, cap `100`)
- - `max_time_ms`: number (default `60000`, cap `600000`)
- - `stream`: boolean
- Example:
- ```json
- { "prompt": "Compare pricing tiers for Supabase, Firebase, and Clerk", "schema": { "type": "object", "properties": { "provider": { "type": "string" }, "free_tier": { "type": "string" }, "paid_start": { "type": "string" } } }, "max_pages": 12 }
- ```
- Tip: output includes a `steps` array showing every action (plan, search, fetch, extract, synthesize) with timings. Use this to debug why an agent run produced a weak result.
- ### diff
- Compare two versions of a page. Point it at a live URL and its cached copy, two URLs, or two markdown blobs.
- Parameters:
- - `old` (object, required): one of `{ url, markdown, content_hash }`
- - `new` (object, required): one of `{ url, markdown }`
- - `output`: `"unified"` (default), `"hunks"`, or `"summary"`
- - `granularity`: `"line"` (default), `"word"`, or `"section"`
- Example:
- ```json
- { "old": { "url": "https://docs.example.com/api" }, "new": { "url": "https://docs.example.com/api" }, "output": "hunks", "granularity": "section" }
- ```
- Tip: same URL on both sides diffs the cached copy against a fresh fetch. Use `output: "summary"` for counts only.
- ### watch
- Monitor a page for changes over time. Lazy execution — checks run when you call `check` or when an overdue job is picked up during another tool run.
- Parameters:
- - `action` (string, required): `"create"`, `"list"`, `"check"`, `"pause"`, `"resume"`, `"delete"`
- - `url` or `urls`: single-URL or batch create (mutually exclusive)
- - `interval_seconds`: required for create (min 60)
- - `selector`: create-only CSS selector to scope the diff
- - `notification`: `"inline"` (default) or an SSRF-guarded webhook URL
- - `job_id`: required for check/pause/resume/delete
- Example:
- ```json
- { "action": "create", "url": "https://nodejs.org/en/blog", "interval_seconds": 21600 }
- ```
- Tip: webhook destinations are SSRF-guarded — a job cannot be pointed at internal or loopback addresses.
- ## Workflow Patterns
- Quick routing:
- - Use when `search` — you need information but don't have a URL.
- - Use when `fetch` — you already have the URL.
- - Use when `crawl` — you need multiple pages from one site.
- - Use when `cache` — you want to check whether something is already on disk.
- - Use when `extract` — you need specific fields, tables, or metadata, not the whole page.
- - Use when `find_similar` — you have a good page/concept and want related content.
- - Use when `research` — a question needs decomposition and multi-source synthesis.
- - Use when `agent` — a natural-language task needs multi-step data gathering.
- - Use when `diff` — you need to see what changed between two versions of a page.
- - Use when `watch` — you want to monitor a page for changes over time.
- **Cache-first lookup.** Before any `fetch` or `search`, probe the cache.
- ```json
- cache({ "query": "oauth2 pkce", "url_pattern": "*auth0.com*" })
- // empty? fall through to search
- search({ "query": "oauth2 pkce flow", "include_domains": ["auth0.com"] })
- ```
- **Fresh content (news, dashboards, changelogs).** Bypass cache explicitly.
- ```json
- search({ "query": "node.js 22 release notes", "force_refresh": true, "time_range": "week" })
- fetch({ "url": "https://nodejs.org/en/blog", "force_refresh": true })
- ```
- **Scoped documentation research.** Crawl the relevant slice, then query cache.
- ```json
- crawl({ "url": "https://docs.astro.build", "strategy": "sitemap", "max_pages": 40 })
- cache({ "query": "server islands hydration", "url_pattern": "*docs.astro.build*" })
- ```
- **Broad exploration.** Pass a query array; dedup is automatic.
- ```json
- search({ "query": ["rust async runtimes comparison", "tokio vs async-std vs smol", "rust executor benchmarks"], "max_results": 8 })
- ```
- **More like this.** Start with a known-good URL, widen via `find_similar`.
- ```json
- find_similar({ "url": "https://react.dev/reference/react/useMemo", "max_results": 6, "include_domains": ["react.dev"] })
- ```
- **Complex synthesis.** One `research` call replaces 5+ manual search/fetch cycles.
- ```json
- research({ "question": "Tradeoffs of vector DBs for RAG at 100M+ embeddings", "depth": "comprehensive" })
- ```
- **Structured data from multiple sources.** Use `agent` with a schema.
- ```json
- agent({ "prompt": "Find latency and pricing for top 5 edge compute providers", "schema": { "type": "object", "properties": { "provider": {"type":"string"}, "cold_start_ms": {"type":"string"}, "price_per_million": {"type":"string"} } } })
- ```
- **Table extraction.** Skip markdown entirely.
- ```json
- extract({ "url": "https://en.wikipedia.org/wiki/List_of_programming_languages", "mode": "tables" })
- ```
- **One-shot structured brief.** Tables + definition lists + JSON-LD + chart hints + key-value pairs in one call.
- ```json
- extract({ "url": "https://example.com/product-page", "mode": "structured" })
- ```
- **Direct quotes with citations.** ML-scored passages are ideal for host-LLM synthesis.
- ```json
- search({ "query": "react server components data fetching", "format": "highlights", "max_highlights": 6, "include_domains": ["react.dev", "nextjs.org"] })
- ```
- ## Parameter Cheat Sheet
- | Situation | Tool + parameters |
- |---|---|
- | Focused lookup, known site | `search` + `max_results: 3` + `include_domains` |
- | Broad topic survey | `search` + `query: [...3-5 variants]` + `max_results: 8` |
- | Fresh content required | any tool + `force_refresh: true` |
- | Doc site indexing | `crawl` + `strategy: "sitemap"` |
- | Site URL inventory only | `crawl` + `strategy: "map"` |
- | Single heading from long page | `fetch` + `section: "..."` |
- | Behind login | `fetch` / `crawl` + `use_auth: true` |
- | Direct answer (sampling client) | `search` + `format: "answer"` |
- | ML-scored passages + citations | `search` + `format: "highlights"` |
- | LLM-ready context blob | `search` + `format: "context"` |
- | Complex question, multi-source | `research` + `depth: "standard"` |
- | Structured multi-page extraction | `agent` + `schema` |
- | One-page structured data | `extract` + `mode: "structured"` (everything) or `"schema"` / `"tables"` (targeted) |
- | Change tracking | `cache` + `check_changes: true` |
- ## Anti-Patterns
- **Do not skip the cache.** Running `search` or `fetch` without probing `cache` wastes time on content already on disk. `research` and `agent` check cache internally; manual `search`/`fetch` do not.
- **Do not send natural-language questions to `search`.** Use keywords. `"how do I debounce in React hooks"` loses to `"react useDebounce hook custom"`.
- **Do not retry an identical failing query.** Reformulate keywords, swap `category`, or add `include_domains`. Same query → same empty result.
- **Do not use `agent` or `research` for one-URL lookups.** Use `fetch`. `agent` is for multi-source gathering; `research` is for decomposable questions.
- **Do not crawl `max_pages: 100` without filters.** Always add `include_patterns` to stay in-scope. Unfiltered crawls fetch nav, footer, and sitemap garbage.
- **Do not fetch whole pages when you need one section.** `fetch` + `section` reads under one heading only.
- **Do not set `force_refresh: true` by default.** It defeats the cache. Use it for news, status, changelogs — content that actually churns.
- **Do not pass a JSON Schema to `extract` without `properties`.** The handler rejects schemas that lack a `properties` key.
- ## CLI Commands
- ```bash
- wigolo # default: start MCP server on stdio
- wigolo mcp # explicit: start MCP server
- wigolo warmup [flags] # install browser engine, bootstrap search engine, optional extras
- wigolo serve # start HTTP daemon on WIGOLO_DAEMON_PORT (default 3333)
- wigolo health # health probe, exits 0 if ok
- wigolo doctor # environment diagnostics
- wigolo auth discover # list CDP sessions (needs WIGOLO_CDP_URL)
- wigolo auth status # show configured auth paths
- wigolo plugin add <git-url> # clone plugin into ~/.wigolo/plugins/
- wigolo plugin list # list installed plugins
- wigolo plugin remove <name> # remove a plugin
- wigolo shell [--json] # interactive REPL against subsystems
- ```
- ## Configuration
- Top environment variables. All optional — defaults are safe.
- | Variable | Default | Purpose |
- |---|---|---|
- | `WIGOLO_DATA_DIR` | `~/.wigolo` | Cache DB, search engine state, plugins, embeddings |
- | `WIGOLO_SEARXNG_URL` | unset | Point at an existing search engine (skips native bootstrap); legacy `SEARXNG_URL` also accepted |
- | `SEARXNG_MODE` | `native` | `native` runs local Python search engine; `docker` runs container |
- | `WIGOLO_CHROME_PROFILE_PATH` | unset | Chrome profile for `use_auth: true` |
- | `WIGOLO_CDP_URL` | unset | Chrome DevTools endpoint (e.g. `http://localhost:9222`) |
- | `MAX_BROWSERS` | `3` | Browser pool size |
- | `WIGOLO_BROWSER_TYPES` | `chromium` | Comma list: `chromium,firefox,webkit` |
- | `WIGOLO_RERANKER` | `none` | `flashrank` for ML reranking |
- | `WIGOLO_EMBEDDING_MODEL` | `BAAI/bge-small-en-v1.5` | Used by `find_similar` |
- | `CACHE_TTL_CONTENT` | `604800` (7d) | Seconds before cached pages expire |
- | `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` |
- Full list: see `src/config.ts`.
- ## Links
- - Repository: https://github.com/KnockOutEZ/wigolo
- - npm: https://www.npmjs.com/package/wigolo
- - License: GNU AGPL-3.0-only — free to use, modify, and self-host; see the root LICENSE
README.md · README · 445 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.
- <div align="center">
- <img alt="wigolo — the go-to web for your agent" src="assets/brand/wigolo-banner.png" width="640">
- Local-first web intelligence for AI agents — **no keys, no cloud, no metered bill.**
- <sub>works with **Claude Code · Cursor · Codex · Gemini CLI · OpenCode · VS Code · Windsurf · Zed · Antigravity**</sub>
- <br>
- <sub>and beyond **LangChain · CrewAI · LlamaIndex · Vercel AI SDK · n8n & self-hosted agents · any MCP client · plain REST**</sub>
- [](https://www.npmjs.com/package/wigolo)
- [](https://www.npmjs.com/package/wigolo)
- [](https://github.com/KnockOutEZ/wigolo/stargazers)
- [](https://github.com/KnockOutEZ/wigolo/actions/workflows/ci.yml)
- [](https://nodejs.org)
- [](https://modelcontextprotocol.io)
- [](#license)
- [](#beta--feedback)
- [](https://x.com/yourtowhid)
- [](https://discord.gg/BkUUgz2bNF)
- <a href="https://trendshift.io/repositories/79424?utm_source=repository-badge&utm_medium=badge&utm_campaign=badge-repository-79424" target="_blank"><img src="https://trendshift.io/api/badge/repositories/79424" alt="wigolo on Trendshift" width="250" height="55"/></a>
- <a href="https://trendshift.io/repositories/79424?utm_source=trendshift-badge&utm_medium=badge&utm_campaign=badge-trendshift-79424" target="_blank" rel="noopener noreferrer"><img src="https://trendshift.io/api/badge/trendshift/repositories/79424/daily?language=TypeScript" alt="KnockOutEZ%2Fwigolo | Trendshift" width="250" height="55"/></a>
- <sub>sponsored by</sub><br>
- <a href="https://wigolo.app/go/testmu/?ref=readme-top"><picture><source media="(prefers-color-scheme: dark)" srcset="assets/sponsors/testmu-ai-dark.svg"><img alt="TestMu AI" src="assets/sponsors/testmu-ai.svg" height="20"></picture></a>
- [Quickstart](#quickstart) · [Tools](#tools) · [Why wigolo](#why-its-different) · [Discord](https://discord.gg/BkUUgz2bNF) · [Sponsors](#sponsors) · [Benchmark](#benchmark) · [Docs](docs/README.md) · [Examples](examples/README.md) · [Licensing](LICENSING.md) · [Feedback](#beta--feedback) · [FAQ](#faq)
- **Join the community on [Discord](https://discord.gg/BkUUgz2bNF)** — questions, help, and what's being built next.
- New features and updates ship steadily. Follow <a href="https://x.com/yourtowhid"><b>@yourtowhid on X</b></a> for all of it and new ways to use wigolo, and reach out there for collaborations or feedback · also on <a href="https://www.linkedin.com/in/yourtowhid/">LinkedIn</a>
- </div>
- ---
- wigolo gives an AI agent one surface for everything web-related: **search, fetch, crawl, extract, cache, find-similar, research,** and autonomous gather loops. It runs wherever your agent runs — as an MCP server next to your coding agent, as a REST/MCP endpoint on the box where your self-hosted agents live, or embedded through an SDK inside your own app. The core tools need no API keys, nothing it touches leaves `~/.wigolo/`, and no bill grows with how much your agent thinks.
- <div align="center">
- <img alt="wigolo demo — Claude Code answering a live web question through wigolo, no API keys" src="assets/wigolo-demo.gif" width="800">
- </div>
- ## Quickstart
- ```bash
- npx wigolo init # set up the local engine — any system
- npx wigolo init --agents=claude-code,cursor # …or set up + wire your day-to-day agents in one command
- ```
- Requires **Node ≥ 20** and ~1.5 GB of free disk on macOS, Linux, or Windows. Bare `init` sets up the local engine: it downloads the browser engine and on-device models, runs a health check, and reports each component. Adding `--agents` wires the named agents in the same run, so a coding agent you use daily is ready in one command.
- - **Supported agents** — `--agents` takes any of `claude-code` · `cursor` · `codex` · `gemini-cli` · `opencode` · `vscode` · `windsurf` · `zed` · `antigravity` (comma-separated); wigolo writes the MCP config and, where supported, instructions for each.
- - **Any other setup** — any MCP client, agent framework, or self-hosted agent registers `npx -y wigolo` in its own MCP config. The [installation guide](docs/installation.md) has the exact config block for every client, plus Docker, Homebrew, and single-file-binary channels.
- - **More on the way** — the supported list keeps growing, and a PR to add your agent is welcome; see [CONTRIBUTING.md](CONTRIBUTING.md).
- - **Interactive setup** — `--interactive` is a plain-text flow; `--wizard` is the full terminal TUI.
- - **Defer downloads** — `--no-warmup` waits until first use. A failed component download never fails setup; init reports what's not ready with the exact fix and still completes.
- `init` is unattended by default, so it's safe in scripts and CI, and any setup problem surfaces right here in the per-component report, before your agent's first call. **Search, fetch, crawl, extract, cache, and find-similar work with no API key.** Check it's healthy anytime:
- ```bash
- npx wigolo doctor
- ```
- To remove everything cleanly, run `npx wigolo config --uninstall --yes`. You can also paste the [installation guide](docs/installation.md) into any AI assistant and let it do the setup; it's written to be self-contained.
- ### Recommended — a free key for `research` & `agent`
- Search, fetch, crawl, extract, cache, and find-similar are **fully keyless**. `research`, `agent`, and `search format=answer` use an LLM to write the synthesized, cited answer. Without one they hand back a raw brief and evidence for your agent to assemble. A free Gemini key turns that into a finished answer:
- ```bash
- export WIGOLO_LLM_PROVIDER=gemini
- export GEMINI_API_KEY=<free-key> # grab one at aistudio.google.com/apikey — the free tier is plenty
- ```
- Any provider works (`anthropic` · `openai` · `groq`), or stay fully local and keyless with `WIGOLO_LLM_PROVIDER=ollama` (or any OpenAI-compatible URL). Set it in your shell or your agent's MCP `env` block. Providers, models, and the keyless local-model ladder are in the [configuration guide](docs/configuration.md).
- ## What your agent gets back
- Every search result is evidence the agent can act on. It carries a verbatim excerpt pinned to its exact position in the source, a citation ID the agent can quote, and a score it can inspect (abridged real shape):
- ```jsonc
- {
- "results": [{
- "title": "Logical replication - PostgreSQL docs",
- "url": "https://www.postgresql.org/docs/current/logical-replication.html",
- "excerpt": "Logical replication is a method of replicating data objects…",
- "citation_id": "src-1",
- "source_span": { "start": 1042, "end": 1305 }, // byte-exact provenance
- "evidence_score": { "final": 0.86, "semantic": 0.91, "lexical": 0.78, "engine_consensus": 3 }
- }],
- "citations": [{ "id": "src-1", "url": "…" }],
- "freshness_signal": { "published": "2026-05-12", "confidence": "high" }
- }
- ```
- Weak results get flagged as junk by wigolo's own scorer. Failed engines are reported and stale cache is labeled, so the agent always knows what it's standing on. Full response contracts per tool are in the [tools reference](docs/tools.md).
- ## Tools
- | Tool | What it does |
- |------|--------------|
- | 🔎 `search` | Multi-engine web search (18 direct adapters) with rank fusion, ML reranking, and an explainable per-result score. Pass a query **array** for parallel breadth. Scope by domain and time range, match an exact phrase, or return image results. |
- | 📄 `fetch` | Load one URL through a tiered router that auto-escalates from plain HTTP to a headless browser engine on anti-bot challenges or SPA shells. Clean markdown + metadata + links. Handles PDFs, a single-heading `section`, authenticated sessions, and page actions (click / type / scroll / screenshot). |
- | 🕸️ `crawl` | Multi-page crawl — BFS, DFS, sitemap, or map-only. Per-domain rate limits, robots.txt respect, boilerplate dedup. |
- | 🧩 `extract` | Structured data from a page: tables, metadata, JSON-LD, brand identity, named schemas (Article / Recipe / Product / …), or any custom JSON Schema. |
- | 💾 `cache` | Query everything already seen — keyword or hybrid semantic. Plus stats, clear, and change detection. |
- | 🧲 `find_similar` | Pages similar to a URL or a concept, via 3-way fusion of keyword + semantic + live web. |
- | 🧠 `research` | Decompose a question → fan out sub-queries → fetch sources → synthesize a cited report (or a structured brief the host LLM writes from). |
- | 🤖 `agent` | Autonomous gather loop: plan → search → fetch → extract → synthesize, with a step log, time budget, and optional output schema. |
- | 🔁 `diff` + ⏱️ `watch` | See exactly what changed on a page since last visit; re-check on demand and deliver changes to a webhook. |
- Every tool also runs from the terminal (`wigolo search "…" --json`), from an interactive shell with NDJSON piping (`wigolo shell`), over REST, and through the SDKs — [CLI reference](docs/cli.md). Per-tool guides with the full parameter set are in [docs/tools.md](docs/tools.md); runnable examples are in [examples/](examples/README.md).
- ## Why it's different
- wigolo isn't a free stand-in for the paid tools — it's built to match them. It's a focused web layer for your agents: an MCP and REST surface they call directly, with the search and extraction quality the paid services charge for. What separates it:
- - **Built for agents.** One MCP call fans out many queries across many engines in parallel, which a serial host tool-loop can't replicate. Every result carries transparent per-result scoring, and output is budget-aware.
- - **Honest output.** Stale cache, failed fetches, degraded backends, and truncation are surfaced in the result. When a bot-protected page can't be read, you get a labeled `blocked_by_challenge` failure, not a challenge shell returned as content.
- - **$0 per query, free to re-query.** Default search talks to public engines through direct adapters; the reranker and embeddings run on-device. Every response is cached, so asking again is instant and costs nothing.
- - **Private by default.** Cache, embeddings, models, and config live under `~/.wigolo/`. Nothing reaches a third party unless you explicitly opt into an LLM for synthesis.
- Here's what one real result looks like, dissected. It includes the failed engine and the weak result, because those are part of the answer too:
- <div align="center">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="assets/promo/anatomy-dark.svg">
- <img alt="Anatomy of a wigolo result: explainable score decomposition, live engine telemetry, surfaced degradation, self-flagged junk — one real query, captured live" src="assets/promo/anatomy.svg" width="880">
- </picture>
- </div>
- ## Benchmark
- > **All four tools converged on the same core answer, and only one of them handed back verbatim, byte-pinned evidence while doing it.**
- One cold query ran live inside a single **Claude Fable 5** session, fanned out to four web tools on equal footing (built-in **WebSearch**, **wigolo**, **Tavily**, **Exa**), and was judged by the agent on the evidence alone. All four converged on the same answer and the same top source, so the parity is demonstrated on-screen. wigolo alone returned verbatim excerpts pinned to byte-offset source spans, an explainable score decomposition, and live per-engine telemetry, and its own scorer flagged two weak results as junk. The cloud tools earn their place too: Exa rendered the official docs' comparison matrix in full. Run your own query and you'll see the same shape.
- <div align="center">
- <img alt="wigolo vs built-in WebSearch, Tavily, and Exa on one real query, driven by Claude Fable 5" src="assets/wigolo-vs.gif" width="900">
- </div>
- ### How it compares
- | | wigolo | Firecrawl | Exa | Tavily |
- |---|:---:|:---:|:---:|:---:|
- | Multi-engine web search | ✅ | ✅ | ✅ | ✅ |
- | Fetch & structured extraction | ✅ | ✅ | ✅ | ✅ |
- | Whole-site crawl & map | ✅ | ✅ | — | ✅ |
- | Verbatim excerpts pinned to byte-offset source spans | ✅ | — | — | — |
- | Explainable per-result score decomposition | ✅ | — | — | — |
- | Persistent local memory — re-query instantly, offline | ✅ | — | — | — |
- | Query data stays on your machine | ✅ | — | — | — |
- | API key / account | none | required | required | required |
- | Cost per query | $0 | metered | metered | metered |
- <sub>Feature standing as of July 2026 — check each vendor's docs for current state.</sub>
- That last row compounds, because agents ask in bursts:
- <div align="center">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="assets/promo/meter-dark.svg">
- <img alt="The meter: a metered cloud API's cost climbs with every query while wigolo stays flat at zero dollars — illustrative pricing" src="assets/promo/meter.svg" width="880">
- </picture>
- </div>
- ## Beyond your editor
- The same ten tools serve every kind of agent, over whichever surface fits: MCP for coding agents, REST for everything else, SDKs to embed, and framework wrappers to drop in.
- ### REST API — `wigolo serve`
- One process exposes a plain-JSON REST API next to the MCP transport. No MCP client needed, just curl:
- ```bash
- wigolo serve # 127.0.0.1:3333 — loopback is open; off-loopback requires a token
- curl -sX POST http://127.0.0.1:3333/v1/search \
- -H 'Content-Type: application/json' \
- -d '{"query":"local-first software","max_results":5}'
- ```
- `POST /v1/{tool}` covers all ten tools, `GET /openapi.json` is the OpenAPI 3.1 contract, and `/mcp` + `/sse` serve remote MCP clients from the same port. Bind past loopback and a bearer token is required, so the server fails closed by default. Point n8n, a Hermes-style assistant, or any self-hosted agent at it. → [REST API](docs/rest-api.md)
- ### SDKs — TypeScript & Python
- Thin, typed clients with an embedded local mode that finds or starts the daemon for you. No separate `serve` step.
- **TypeScript** — `npm install wigolo-sdk` (zero-dep; Node / Bun / Deno / edge):
- ```ts
- import { createLocalClient } from 'wigolo-sdk/local';
- const { client, close } = await createLocalClient(); // reuse a running daemon, or spawn one
- const res = await client.search({ query: 'local-first web search', max_results: 5 });
- console.log(res.results.map((r) => r.title));
- await close(); // stops the daemon only if this call spawned it
- ```
- **Python** — `pip install wigolo` (standard library only; sync + async):
- ```python
- from wigolo import local_client
- with local_client() as client: # reuse a healthy daemon, or spawn one
- res = client.search(query="local-first web search", max_results=5)
- for r in res["results"]:
- print(r["title"], r["url"])
- ```
- → [SDKs & embedded mode](docs/sdks.md)
- ### Framework integrations
- Drop wigolo's tools into the framework you already use. You get the full ten-tool surface, including the cache / find_similar / research / agent that most framework web-tools don't ship:
- | Framework | Package | What you get |
- |-----------|---------|--------------|
- | **LangChain** | `wigolo-langchain` | each tool as a `BaseTool`, plus a `BaseRetriever` over search / find_similar for RAG |
- | **CrewAI** | `wigolo-crewai` | `wigolo_tools()` → hand the set to any crew |
- | **LlamaIndex** | `wigolo-llamaindex` | a `BaseReader` that loads fetched / crawled / searched pages as documents |
- | **Vercel AI SDK** | `wigolo-vercel-ai-sdk` | tool factories for `generateText` / `streamText`, edge-friendly |
- → [Framework integrations](docs/sdks.md)
- ### Docker
- ```bash
- # stdio MCP — wire it into any MCP client as command: docker
- docker run -i --rm -v wigolo-data:/data ghcr.io/knockoutez/wigolo
- # HTTP server for remote / multi-client use
- docker run -p 3333:3333 -v wigolo-data:/data \
- -e WIGOLO_API_TOKEN=a-long-random-secret \
- ghcr.io/knockoutez/wigolo serve --host 0.0.0.0
- ```
- The slim image lazy-loads models into the volume; `ghcr.io/knockoutez/wigolo:latest-full` preinstalls the browser engine. Versioned releases use the same `-full` suffix (for example, `0.3.1-full`). Also on Docker Hub as `towhid69420/wigolo`. → [installation & all channels](docs/installation.md)
- ### Agent skills
- An 11-pack skill catalog teaches your coding agent to drive each tool well. It's installed by `init` and managed with `wigolo skills add|list|remove`. → [skills](docs/skills.md)
- One note for self-hosters: some challenge-protected sites score IP reputation, so a datacenter IP won't clear walls a home connection would. wigolo labels those failures, and the [self-hosting guide](docs/self-hosting.md) covers the opt-in proxy answer.
- ## Star history
- <div align="center">
- <a href="https://www.star-history.com/#KnockOutEZ/wigolo&Date">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/KnockOutEZ/wigolo/star-chart/star-history-dark.svg">
- <img alt="wigolo GitHub stars over time" src="https://raw.githubusercontent.com/KnockOutEZ/wigolo/star-chart/star-history.svg" width="880">
- </picture>
- </a>
- <sub>Refreshed daily from the GitHub API. <a href="https://github.com/KnockOutEZ/wigolo">Add a ⭐</a> if wigolo is useful to you.</sub>
- </div>
- ## Architecture
- A single Node process speaks MCP (JSON-RPC over stdio). Everything heavy is local and lazy-loaded, so a zero-key install pays nothing for the parts it isn't using.
- ```mermaid
- flowchart TD
- A["🤖 AI agent<br/>any MCP client · REST · SDK"]
- A -->|MCP over stdio| B["<b>wigolo</b><br/>10 tools · dynamic instructions<br/>in-process browser pool + cache + models"]
- B --> C{"Tool layer"}
- C --> T1["search · fetch · crawl · extract"]
- C --> T2["cache · find_similar · research · agent"]
- T1 --> F["⚙️ Fetch router<br/>tiered escalation, learned per domain"]
- T1 --> S["⚙️ Search<br/>18 engines → rank fusion → ML rerank<br/><i>explainable evidence score</i>"]
- T2 --> DB[("🗄️ Local cache<br/>keyword + vector index")]
- T2 --> ML["🧠 On-device ML<br/>embeddings + reranker"]
- F -.->|optional| LLM["☁️ LLM<br/>synthesis only · opt-in"]
- S -.->|optional| SX["🔀 Aggregator backend<br/>opt-in legacy / hybrid"]
- F --> WEB["🌍 Public web"]
- S --> WEB
- style B fill:#7c3aed,stroke:#5b21b6,color:#fff
- style WEB fill:#0ea5e9,stroke:#0369a1,color:#fff
- style DB fill:#1e293b,stroke:#334155,color:#fff
- style LLM stroke-dasharray: 5 5
- style SX stroke-dasharray: 5 5
- ```
- - **Code beats model.** Deterministic work stays off the LLM: canonicalization, rank fusion, dedup, and schema matching. The model is reserved for judgment, opt-in, and capped per request. LLM-filled fields are checked against the source and nulled if absent.
- - **Signal-driven routing.** The fetch ladder escalates to a real browser on observable signals, not domain guesses: SPA markers, challenge bodies, thin content. It learns per domain, unlearns when a site stops needing it, and `wigolo tune list` shows you exactly what it learned.
- - **Reads pages the way a browser does.** Tiered fetching waits out interstitial challenges and reuses clearances per domain, politely: robots.txt respected, per-domain rate limits, research-grade volumes. When a wall stays up, the failure is labeled and reported.
- ## Configuration
- A clean install works out of the box. Three settings raise output quality:
- ```bash
- # 1. Synthesis — the biggest lever (research / agent / search-answer write real prose)
- export WIGOLO_LLM_PROVIDER=gemini # or anthropic / openai / groq / ollama (keyless)
- export GEMINI_API_KEY=<your-key>
- # 2. Wider retrieval funnel
- export WIGOLO_SEARCH=hybrid # core engines + aggregator fallback
- export WIGOLO_GITHUB_TOKEN=... # GitHub code search 10 → 30 req/min
- # 3. Land more fetches, stay warm
- export WIGOLO_TLS_TIER=auto # per-domain learned fetch hardening
- export WIGOLO_EAGER_WARMUP=1 # pay the ~1s model load up front
- ```
- **Per-call habits that pay off:** query **arrays** (`["a","b","c"]`) for parallel breadth · `search_depth: "deep"` for queries that matter · `include_domains` as a hard filter for docs lookups. The full reference covers every environment variable, config-file key, search backend, cache TTL, and serve limit; it's in the [configuration guide](docs/configuration.md).
- ## Docs & examples
- **[docs/](docs/README.md)** — the complete manual:
- [getting started](docs/getting-started.md) · [installation & channels](docs/installation.md) · [configuration](docs/configuration.md) · [tools reference](docs/tools.md) · [CLI & shell](docs/cli.md) · [REST API](docs/rest-api.md) · [SDKs & integrations](docs/sdks.md) · [self-hosting](docs/self-hosting.md) · [agent skills](docs/skills.md) · [plugins](docs/plugins.md) · [troubleshooting & FAQ](docs/troubleshooting.md) · [privacy & security](docs/privacy-security.md)
- **[examples/](examples/README.md)** — runnable, each with a README (and most with a terminal recording): one-shot CLI, NDJSON shell pipelines, REST via curl, TypeScript & Python SDKs, Vercel AI SDK tools, pointing self-hosted n8n at a remote wigolo, watch-with-webhook, and writing your own search-engine plugin. The docs are also rendered on the site at **[wigolo.app/docs](https://wigolo.app/docs/)**.
- ## Beta & feedback
- wigolo is in **public beta**. Everything documented here works and is held to a 7,600-test suite; it's stable, and beta is about the polish bar. It stays beta until enough people have used it, kicked it, and starred it that calling it v1 means something. Your feedback shapes what comes next, and every report is read, usually the same day:
- - 🐛 **[Report a bug](https://github.com/KnockOutEZ/wigolo/issues/new?template=bug_report.yml)** — broke, misbehaved, surprised you
- - 💡 **[Request a feature](https://github.com/KnockOutEZ/wigolo/issues/new?template=feature_request.yml)** — something it should do
- - 💬 **[Ask anything](https://github.com/KnockOutEZ/wigolo/discussions)** — questions, setups, show & tell
- If wigolo earns a place in your setup, three things keep it going: a ⭐ **star** (it's how open source gets found), a **[☕ coffee](https://buymeacoffee.com/knockoutez)** (the open-source version is the full product; commercial licenses fund it, not gated features), or **[an email](mailto:ktowhid20@gmail.com)** that goes straight to the one developer who wrote the code.
- ## Troubleshooting
- `wigolo doctor` names any broken component and the exact env var or command that fixes it; `wigolo doctor --fix` repairs the common cases, and `wigolo verify` health-checks every component. A component failing during `init` doesn't break wigolo: `init` still exits 0, and core search / fetch / crawl / extract / cache work with no models and no browser. Quick hits:
- - **Slow or failed downloads** — re-run `wigolo warmup --all` (or `--browser` / `--embeddings` / `--reranker`); they resume and retry.
- - **Browser won't launch on Linux** — `wigolo warmup --browser` installs the OS libraries (or prints the exact command).
- - **Native build error / unusual Node** — use an LTS — **Node 22 or 24** — or Node 26 (supported; Current until it becomes LTS).
- - **Behind a proxy** — `USE_PROXY=true` + `PROXY_URL`; add `NODE_EXTRA_CA_CERTS` for TLS-inspecting proxies.
- - **Your agent asks permission on every call** — allow the tools in your client, then restart it; rules are read at session start. [Details](docs/troubleshooting.md#your-agent-keeps-asking-permission).
- The full guide covers per-symptom fixes, a "what still works when X fails" map, platform notes (incl. linux-arm64), and offline installs: **[docs/troubleshooting.md](docs/troubleshooting.md)**.
- ## FAQ
- <details>
- <summary><b>Free? What's the catch?</b></summary>
- No catch by design. The expensive parts (ranking, embeddings, the browser engine) run on *your* hardware, so there's no per-query cost to recover and no reason for a meter. It's funded by commercial licenses and sponsors, and the AGPL legally prevents a switch into a closed hosted product.
- </details>
- <details>
- <summary><b>Is the quality really on par with the paid services?</b></summary>
- The benchmark section above is a live 4-way run you can reproduce: everyday agent queries land at parity, the paid tools still win some deep-extraction edge cases, and crawling is where wigolo is strongest. Every result shows its scoring, so you don't have to take anyone's word for it.
- </details>
- <details>
- <summary><b>Won't public search engines block or rot?</b></summary>
- It's engineered for exactly that: 18 engines fused with rank fusion (any one failing barely moves results), a tiered fetch ladder with per-domain learning, and an optional aggregator fallback. Degraded backends are *reported in the output*, and the local cache means everything already seen keeps working regardless.
- </details>
- <details>
- <summary><b>Is this kind of scraping OK?</b></summary>
- wigolo reads the public web the way a browser does: robots.txt respected by default, per-domain rate limits, and research-grade volumes for one agent on one machine. It sits deliberately at the polite end of the spectrum.
- </details>
- <details>
- <summary><b>Can I use this at work?</b></summary>
- Yes. If you're a company or building a commercial product on wigolo, please take a commercial license or sponsor the project. See [LICENSING.md](LICENSING.md).
- </details>
- <details>
- <summary><b>Why 1.5 GB of disk?</b></summary>
- That's the on-device brain: a full browser engine plus the ranking and embedding models the cloud services run on their side and bill you for. Once it's on disk, every query uses it for free.
- </details>
- ## Available on
- - **npm** — [`wigolo`](https://www.npmjs.com/package/wigolo) *(primary channel — the Quickstart above)*
- - **PyPI** — [`wigolo`](https://pypi.org/project/wigolo/) *(Python SDK)*
- - **Docker** — [`ghcr.io/knockoutez/wigolo`](https://github.com/KnockOutEZ/wigolo/pkgs/container/wigolo) · [`towhid69420/wigolo`](https://hub.docker.com/r/towhid69420/wigolo)
- - **Official MCP Registry** — `io.github.KnockOutEZ/wigolo`
- - **Directories** — [Glama](https://glama.ai/mcp/servers/KnockOutEZ/wigolo) · [Smithery](https://smithery.ai/server/ktowhid20/wigolo) · [mcp.so](https://mcp.so/server/wigolo/KnockOutEZ) · [LobeHub](https://lobehub.com/mcp/knockoutez-wigolo)
- Homebrew, `curl | sh`, and the single-file binary are covered in the [installation guide](docs/installation.md). Use one channel per machine; they all share `~/.wigolo`.
- ## Sponsors
- Thank you to the sponsors below, who help keep wigolo maintained and free for everyone to use. Their support goes straight into the work.
- <div align="center">
- <a href="https://wigolo.app/go/testmu/?ref=readme">
- <picture>
- <source media="(prefers-color-scheme: dark)" srcset="assets/sponsors/testmu-ai-dark.svg">
- <img alt="TestMu AI" src="assets/sponsors/testmu-ai.svg" height="32">
- </picture>
- </a>
- <sub>**[TestMu AI](https://wigolo.app/go/testmu/?ref=readme)** (formerly LambdaTest) is the world's first full-stack agentic AI quality engineering platform, trusted by 18,000+ enterprises.</sub>
- </div>
- **wigolo is free for everyone.** If you or your company would like to help keep it maintained, there's room for more sponsors — reach out at **[ktowhid20@gmail.com](mailto:ktowhid20@gmail.com)**, or see [what sponsorship includes](https://wigolo.app/sponsors/) and the terms in [SPONSORS.md](SPONSORS.md). A one-off via [Buy Me a Coffee](https://buymeacoffee.com/knockoutez) is welcome too.
- ## Contributing
- Bug reports, feature requests, and PRs are all welcome; see **[CONTRIBUTING.md](CONTRIBUTING.md)**. Keep tool handlers thin, add tests, and run the suite before opening a PR. The friendliest entry point is the plugin system for custom search engines and extractors: [add a search engine in ~100 lines](docs/plugins.md), with a template in [`examples/plugin-search-engine`](examples/plugin-search-engine).
- ## License
- Free and open source under the **[GNU AGPL-3.0](LICENSE)**, and it'll stay that way. If you're a company, or you're making money with wigolo, please take a commercial license or sponsor the project. That's what keeps it maintained. See **[LICENSING.md](LICENSING.md)**.
- See **[SECURITY.md](SECURITY.md)** to report a vulnerability and **[TRADEMARK.md](TRADEMARK.md)** for use of the name.
- <div align="center">
- <br>
- wigolo is open source, free, and actively maintained.
- If it saves you a metered search bill, a ⭐, a sharp issue, or a **[☕ coffee](https://buymeacoffee.com/knockoutez)** helps keep it sustainable.
- <sub>Built and maintained by <a href="https://github.com/KnockOutEZ">@KnockOutEZ</a> · <a href="mailto:ktowhid20@gmail.com">ktowhid20@gmail.com</a> · <a href="https://x.com/yourtowhid">X</a> · <a href="https://www.linkedin.com/in/yourtowhid/">LinkedIn</a></sub>
- </div>
packaging/install.sh · Bundled script · 401 lines · 1 finding marked
- #!/bin/sh
- # wigolo bootstrap installer — local-first web intelligence for AI agents.
- #
- # Usage:
- # curl -fsSL https://wigolo.dev/install.sh | sh
- Note: Pipes a downloaded script directly into a shell (curl | sh / wget | bash)
- Line 5 on GitHub
- # sh install.sh --uninstall
- #
- # What it does:
- # 1. Detects your OS + CPU and bails early (with guidance) on unsupported ones.
- # 2. Installs wigolo under ~/.wigolo/ using a pinned, checksum-verified
- # language runtime — no system packages touched, no root required.
- # 3. Writes a launcher at ~/.wigolo/bin/wigolo and prints how to wire it into
- # your AI agent.
- #
- # This does NOT need any API keys for core work. Optional cloud keys add
- # answer synthesis; configure them later with `wigolo config`.
- #
- # Environment overrides (advanced / testing):
- # WIGOLO_INSTALL_PREFIX install root (default: $HOME/.wigolo)
- # WIGOLO_INSTALL_SOURCE install from a local package tarball instead of the
- # registry (absolute path to a `npm pack` .tgz)
- # HTTPS_PROXY/https_proxy honored for all downloads
- #
- # POSIX sh. set -eu (no pipefail — not POSIX).
- set -eu
- # ---------------------------------------------------------------------------
- # Pinned language runtime. Bump deliberately; both must move together.
- # ---------------------------------------------------------------------------
- NODE_VERSION="v22.23.1"
- NODE_DIST_BASE="https://nodejs.org/dist"
- PKG_NAME="wigolo"
- # Canonical release location for future prebuilt binaries (not yet published;
- # the runtime path below is the live path until a release ships them).
- BIN_RELEASE_BASE="https://github.com/KnockOutEZ/wigolo/releases/latest/download"
- # ---------------------------------------------------------------------------
- # Paths
- # ---------------------------------------------------------------------------
- PREFIX="${WIGOLO_INSTALL_PREFIX:-$HOME/.wigolo}"
- BIN_DIR="$PREFIX/bin"
- TOOL_DIR="$PREFIX/tool"
- RUNTIME_DIR="$PREFIX/runtime"
- TMP_DIR=""
- # ---------------------------------------------------------------------------
- # Output helpers (all to stderr so a piped install can stay quiet on stdout)
- # ---------------------------------------------------------------------------
- info() { printf '%s\n' "$*" >&2; }
- step() { printf '\n==> %s\n' "$*" >&2; }
- warn() { printf 'warning: %s\n' "$*" >&2; }
- die() {
- printf 'error: %s\n' "$*" >&2
- exit 1
- }
- cleanup() {
- if [ -n "$TMP_DIR" ] && [ -d "$TMP_DIR" ]; then
- rm -rf "$TMP_DIR"
- fi
- }
- trap cleanup EXIT INT TERM
- # ---------------------------------------------------------------------------
- # Download abstraction — curl or wget, HTTPS_PROXY honored automatically by
- # both tools via the standard env vars.
- # ---------------------------------------------------------------------------
- DL_TOOL=""
- detect_downloader() {
- if command -v curl >/dev/null 2>&1; then
- DL_TOOL="curl"
- elif command -v wget >/dev/null 2>&1; then
- DL_TOOL="wget"
- else
- die "neither curl nor wget found. Install one, or use the npm channel:
- npm install -g $PKG_NAME"
- fi
- }
- # download <url> <dest-file>
- download() {
- url="$1"
- dest="$2"
- if [ "$DL_TOOL" = "curl" ]; then
- curl -fSL --retry 3 -o "$dest" "$url" || die "download failed: $url"
- else
- wget -q -O "$dest" "$url" || die "download failed: $url"
- fi
- }
- # ---------------------------------------------------------------------------
- # sha256 — fail closed if no tool is available.
- # ---------------------------------------------------------------------------
- sha256_of() {
- file="$1"
- if command -v sha256sum >/dev/null 2>&1; then
- sha256sum "$file" | awk '{print $1}'
- elif command -v shasum >/dev/null 2>&1; then
- shasum -a 256 "$file" | awk '{print $1}'
- else
- die "no sha256 tool (sha256sum/shasum) found — cannot verify downloads. Aborting."
- fi
- }
- # verify_sha256 <file> <expected-hex>
- verify_sha256() {
- file="$1"
- expected="$2"
- [ -n "$expected" ] || die "empty expected checksum — refusing to trust $file"
- actual="$(sha256_of "$file")"
- if [ "$actual" != "$expected" ]; then
- die "checksum mismatch for $(basename "$file")
- expected: $expected
- actual: $actual"
- fi
- }
- # ---------------------------------------------------------------------------
- # Platform detection
- # ---------------------------------------------------------------------------
- OS=""
- ARCH=""
- detect_platform() {
- uname_s="$(uname -s 2>/dev/null || echo unknown)"
- uname_m="$(uname -m 2>/dev/null || echo unknown)"
- case "$uname_s" in
- Linux) OS="linux" ;;
- Darwin) OS="darwin" ;;
- MINGW* | MSYS* | CYGWIN* | Windows_NT)
- die "Windows is not supported by this installer. Use the npm channel:
- npm install -g $PKG_NAME"
- ;;
- *)
- die "unsupported OS: $uname_s. Use the npm channel:
- npm install -g $PKG_NAME"
- ;;
- esac
- case "$uname_m" in
- x86_64 | amd64) ARCH="x64" ;;
- arm64 | aarch64) ARCH="arm64" ;;
- armv7* | armv6*)
- die "32-bit ARM ($uname_m) is not supported. Try Docker, or the npm channel:
- npm install -g $PKG_NAME"
- ;;
- riscv*)
- die "RISC-V ($uname_m) is not supported. Try Docker, or the npm channel:
- npm install -g $PKG_NAME"
- ;;
- *)
- die "unsupported CPU architecture: $uname_m. Try Docker, or the npm channel:
- npm install -g $PKG_NAME"
- ;;
- esac
- if [ "$OS" = "linux" ]; then
- check_linux_libc
- fi
- }
- # musl / Alpine and old-glibc bail — the pinned runtime and native components
- # are built against modern glibc.
- check_linux_libc() {
- if [ -f /etc/alpine-release ]; then
- die "Alpine / musl libc is not supported by the prebuilt runtime.
- Use Docker (glibc-based image) or the npm channel:
- npm install -g $PKG_NAME"
- fi
- # ldd --version prints the libc flavor + version to stdout or stderr.
- ldd_out=""
- if command -v ldd >/dev/null 2>&1; then
- ldd_out="$(ldd --version 2>&1 || true)"
- fi
- case "$ldd_out" in
- *musl*)
- die "musl libc detected. Use Docker (glibc-based image) or the npm channel:
- npm install -g $PKG_NAME"
- ;;
- esac
- # Require glibc >= 2.28 (the runtime's build baseline).
- glibc_line="$(printf '%s\n' "$ldd_out" | head -n 1)"
- glibc_ver="$(printf '%s\n' "$glibc_line" | grep -oE '[0-9]+\.[0-9]+' | head -n 1)"
- if [ -n "$glibc_ver" ]; then
- glibc_major="${glibc_ver%%.*}"
- glibc_minor="${glibc_ver#*.}"
- if [ "$glibc_major" -lt 2 ] || { [ "$glibc_major" -eq 2 ] && [ "$glibc_minor" -lt 28 ]; }; then
- die "glibc $glibc_ver is too old (need >= 2.28). Use Docker or the npm channel:
- npm install -g $PKG_NAME"
- fi
- fi
- }
- # ---------------------------------------------------------------------------
- # Runtime install — download pinned tarball, verify against SHASUMS256.txt,
- # unpack under $RUNTIME_DIR.
- # ---------------------------------------------------------------------------
- NODE_BIN=""
- NPM_BIN=""
- install_runtime() {
- node_pkg="node-$NODE_VERSION-$OS-$ARCH"
- tarball="$node_pkg.tar.gz"
- tar_url="$NODE_DIST_BASE/$NODE_VERSION/$tarball"
- sums_url="$NODE_DIST_BASE/$NODE_VERSION/SHASUMS256.txt"
- # Already installed and correct version? Skip re-download on upgrade re-runs.
- if [ -x "$RUNTIME_DIR/bin/node" ]; then
- have="$("$RUNTIME_DIR/bin/node" --version 2>/dev/null || echo none)"
- if [ "$have" = "$NODE_VERSION" ]; then
- info "Runtime $NODE_VERSION already present — reusing."
- NODE_BIN="$RUNTIME_DIR/bin/node"
- NPM_BIN="$RUNTIME_DIR/bin/npm"
- return 0
- fi
- fi
- step "Downloading the bundled runtime ($node_pkg)"
- download "$tar_url" "$TMP_DIR/$tarball"
- step "Verifying runtime checksum (fail-closed)"
- download "$sums_url" "$TMP_DIR/SHASUMS256.txt"
- expected="$(grep " $tarball\$" "$TMP_DIR/SHASUMS256.txt" | awk '{print $1}')"
- [ -n "$expected" ] || die "no checksum entry for $tarball in SHASUMS256.txt"
- verify_sha256 "$TMP_DIR/$tarball" "$expected"
- info "Checksum OK."
- step "Unpacking the runtime into $RUNTIME_DIR"
- rm -rf "$RUNTIME_DIR"
- mkdir -p "$RUNTIME_DIR"
- # Strip the top-level node-<ver>-<os>-<arch>/ directory.
- tar -xzf "$TMP_DIR/$tarball" -C "$RUNTIME_DIR" --strip-components=1
- NODE_BIN="$RUNTIME_DIR/bin/node"
- NPM_BIN="$RUNTIME_DIR/bin/npm"
- [ -x "$NODE_BIN" ] || die "runtime unpack failed — $NODE_BIN missing"
- }
- # ---------------------------------------------------------------------------
- # Tool install — use the PINNED runtime's own package manager so any native
- # components resolve prebuilds for the correct runtime, into $TOOL_DIR.
- # ---------------------------------------------------------------------------
- install_tool() {
- mkdir -p "$TOOL_DIR"
- spec="$PKG_NAME"
- if [ -n "${WIGOLO_INSTALL_SOURCE:-}" ]; then
- [ -f "$WIGOLO_INSTALL_SOURCE" ] || die "WIGOLO_INSTALL_SOURCE not found: $WIGOLO_INSTALL_SOURCE"
- spec="$WIGOLO_INSTALL_SOURCE"
- step "Installing wigolo from local package: $spec"
- else
- step "Installing wigolo from the registry"
- fi
- # PATH so the runtime's node is what npm and node-gyp see.
- NPM_CONFIG_PREFIX="$TOOL_DIR" \
- PATH="$RUNTIME_DIR/bin:$PATH" \
- "$NPM_BIN" install -g --no-fund --no-audit "$spec" >&2 \
- || die "tool install failed"
- }
- # ---------------------------------------------------------------------------
- # Prebuilt binary path (canonical when a release ships one). Returns 0 if a
- # binary was installed, 1 if none is available for this platform.
- # ---------------------------------------------------------------------------
- try_prebuilt_binary() {
- # Skip entirely when installing from a local tarball (test / dev path).
- [ -z "${WIGOLO_INSTALL_SOURCE:-}" ] || return 1
- bin_name="wigolo-$OS-$ARCH"
- bin_url="$BIN_RELEASE_BASE/$bin_name"
- sums_url="$BIN_RELEASE_BASE/checksums.txt"
- # Probe the checksums file first; absence => no binary channel yet.
- if [ "$DL_TOOL" = "curl" ]; then
- curl -fsSL --retry 2 -o "$TMP_DIR/checksums.txt" "$sums_url" 2>/dev/null || return 1
- else
- wget -q -O "$TMP_DIR/checksums.txt" "$sums_url" 2>/dev/null || return 1
- fi
- expected="$(grep " $bin_name\$" "$TMP_DIR/checksums.txt" | awk '{print $1}')"
- [ -n "$expected" ] || return 1
- step "Downloading prebuilt binary ($bin_name)"
- download "$bin_url" "$TMP_DIR/$bin_name" || return 1
- step "Verifying binary checksum (fail-closed)"
- verify_sha256 "$TMP_DIR/$bin_name" "$expected"
- mkdir -p "$BIN_DIR"
- mv "$TMP_DIR/$bin_name" "$BIN_DIR/wigolo"
- chmod 0755 "$BIN_DIR/wigolo"
- info "Installed prebuilt binary to $BIN_DIR/wigolo"
- return 0
- }
- # ---------------------------------------------------------------------------
- # Shim — prepend the runtime's bin to PATH so the CLI's own tooling probes
- # (e.g. resolving a package runner for the browser engine) work, then exec the
- # installed CLI.
- # ---------------------------------------------------------------------------
- write_shim() {
- mkdir -p "$BIN_DIR"
- entry="$TOOL_DIR/lib/node_modules/$PKG_NAME/dist/index.js"
- [ -f "$entry" ] || die "CLI entry not found at $entry after install"
- cat > "$BIN_DIR/wigolo" <<EOF
- #!/bin/sh
- # wigolo launcher (generated by install.sh). Prepends the bundled runtime so
- # the CLI's package-runner probes resolve, then runs the installed CLI.
- export PATH="$RUNTIME_DIR/bin:\$PATH"
- exec "$RUNTIME_DIR/bin/node" "$TOOL_DIR/lib/node_modules/$PKG_NAME/dist/index.js" "\$@"
- EOF
- chmod 0755 "$BIN_DIR/wigolo"
- }
- # ---------------------------------------------------------------------------
- # Uninstall — remove the tool, but PRESERVE user data (cache/models/keys).
- # ---------------------------------------------------------------------------
- do_uninstall() {
- step "Removing the wigolo tool (preserving your data in $PREFIX)"
- rm -rf "$BIN_DIR" "$TOOL_DIR" "$RUNTIME_DIR"
- info "Removed: $BIN_DIR, $TOOL_DIR, $RUNTIME_DIR"
- info "Preserved: your cache, models, and keys under $PREFIX"
- info ""
- info "To wipe everything including data: rm -rf $PREFIX"
- exit 0
- }
- # ---------------------------------------------------------------------------
- # Closing message + doctor verdict.
- # ---------------------------------------------------------------------------
- finish() {
- step "Verifying the install"
- if "$BIN_DIR/wigolo" doctor >&2; then
- doctor_ok=1
- else
- doctor_ok=0
- fi
- info ""
- info "wigolo installed to $BIN_DIR/wigolo"
- info ""
- info "Add to your shell (so 'wigolo' resolves in your terminal):"
- info " export PATH=\"$BIN_DIR:\$PATH\""
- info ""
- info "Wire it into your AI agent (use the ABSOLUTE path — MCP clients do not"
- info "read your shell PATH):"
- info " claude mcp add wigolo -- $BIN_DIR/wigolo"
- info ""
- if [ "$doctor_ok" -eq 1 ]; then
- info "Health check: OK."
- else
- info "Health check reported issues above — run '$BIN_DIR/wigolo doctor' for detail."
- fi
- }
- # ---------------------------------------------------------------------------
- # Main
- # ---------------------------------------------------------------------------
- main() {
- for arg in "$@"; do
- case "$arg" in
- --uninstall)
- do_uninstall
- ;;
- -h | --help)
- info "wigolo installer"
- info " (no args) install or upgrade wigolo under $PREFIX"
- info " --uninstall remove the tool, preserving your cache/models/keys"
- exit 0
- ;;
- *)
- die "unknown argument: $arg (try --help)"
- ;;
- esac
- done
- detect_downloader
- detect_platform
- mkdir -p "$PREFIX" "$BIN_DIR"
- TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wigolo-install.XXXXXX")"
- step "Installing wigolo for $OS/$ARCH into $PREFIX"
- if try_prebuilt_binary; then
- info "Using prebuilt binary."
- else
- install_runtime
- install_tool
- write_shim
- fi
- finish
- }
- main "$@"
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.