Skip to content
Back to search

agent-search

Self-hosted search API + MCP server for AI agents. Bundles SearXNG. Zero API keys, one-command deploy. Open-source alternative to Tavily, Exa, and Serper.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/brcrusoe72/agent-search with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/brcrusoe72/agent-search'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/brcrusoe72--agent-search/badge.svg)](/listings/brcrusoe72--agent-search)

HTML

<a href="/listings/brcrusoe72--agent-search"><img src="/api/listings/brcrusoe72--agent-search/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 539 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # AgentSearch
  2. Self-hosted search API for AI agents. Layered content extraction with isolated browser rendering. Optional Tor-anonymized stack. No third-party search API keys, no per-query fees, no vendor lock-in. Optional local bearer auth is supported.
  3. [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) [![PyPI](https://img.shields.io/pypi/v/agentsearch-client)](https://pypi.org/project/agentsearch-client/)
  4. ```bash
  5. git clone https://github.com/brcrusoe72/agent-search.git
  6. cd agent-search
  7. ./scripts/prepare-searxng.sh
  8. docker compose up -d
  9. curl "http://localhost:3939/search?q=distributed+consensus+algorithms"
  10. ```
  11. You now have a deduplicated, multi-engine search API running on `:3939`.
  12. To enable auth on the standard stack, set a token and recreate the API so it reads the new environment. Keep the token in your private deployment environment for subsequent Compose commands. Pass it on all endpoints except `/health`, `/live`, and `/ready`:
  13. ```bash
  14. export AGENT_SEARCH_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"
  15. docker compose up -d --no-deps api
  16. curl -H "Authorization: Bearer $AGENT_SEARCH_TOKEN" \
  17. "http://localhost:3939/search?q=distributed+consensus+algorithms"
  18. ```
  19. For an existing private stack, use the same Compose files and project name used at startup, and recreate both `api` and `api-private` with the token set. Setting a client token alone does not enable server authentication.
  20. Prefer not to use Docker for the API server?
  21. ```bash
  22. git clone https://github.com/brcrusoe72/agent-search.git
  23. cd agent-search
  24. ./scripts/install-native.sh
  25. ./scripts/run-native.sh
  26. ```
  27. Native mode requires Python 3.11+ and a reachable SearXNG instance with JSON output enabled. It stores AgentSearch state in `./data`. See [Native Install](docs/native-install.md).
  28. Existing Docker installations: follow [the upgrade procedure](docs/upgrading.md) for data ownership, network changes, and migration of existing SearXNG settings.
  29. ## Verify
  30. Run the self-contained test suite:
  31. ```bash
  32. python -m venv .venv
  33. . .venv/bin/activate
  34. pip install -r requirements.txt pytest requests build wheel "setuptools>=68"
  35. pip install -e sdk
  36. # MCP has a separate Pydantic requirement; keep it isolated.
  37. python -m venv .venv-mcp
  38. .venv-mcp/bin/python -m pip install -e mcp-server pytest
  39. .venv-mcp/bin/python -m pytest tests/test_mcp_contract.py -q
  40. .venv-mcp/bin/python -m pip check
  41. ./scripts/prepare-searxng.sh
  42. pytest tests -q
  43. python -m compileall app adapters mcp-server/agent_search_mcp scripts sdk -q
  44. python -m pip check
  45. ```
  46. The self-contained tests mock SearXNG and do not require Docker or a running local service. Validate the container configuration and pinned engine templates separately with Docker available:
  47. ```bash
  48. docker compose -f docker-compose.yml config --quiet
  49. docker compose -f docker-compose.yml -f examples/compose.private.yml config --quiet
  50. docker build -t agent-search-api:test .
  51. python scripts/check_searxng_config.py
  52. ```
  53. Run the optional live localhost check:
  54. ```bash
  55. AGENTSEARCH_INTEGRATION=1 pytest tests -q
  56. ```
  57. If your local instance requires auth:
  58. ```bash
  59. AGENT_SEARCH_TOKEN="change-me" AGENTSEARCH_INTEGRATION=1 pytest tests -q
  60. ```
  61. Run the optional Docker smoke tests against a running direct/private stack:
  62. ```bash
  63. AGENT_SEARCH_TOKEN="change-me" \
  64. AGENTSEARCH_DOCKER_INTEGRATION=1 \
  65. pytest tests/test_live_docker.py -q
  66. ```
  67. For a small, on-demand live pilot, use the eight fixed cases covering general,
  68. code, academic, and news search. Set `AGENT_SEARCH_TOKEN` when auth is enabled:
  69. ```bash
  70. python scripts/check_search_pilot.py benchmark --label YOUR_DEPLOYED_COMMIT-direct --output /tmp/search-pilot.json
  71. python scripts/check_search_pilot.py reliability --seconds 300 --label YOUR_DEPLOYED_COMMIT-direct --output /tmp/search-reliability.json
  72. ```
  73. Use `--base-url http://127.0.0.1:3940` and a separate output/label for Tor.
  74. The benchmark makes eight search requests and records cache state, timing,
  75. upstream diagnostics, and the first three results. Academic targets use paper
  76. identifiers; news needs human relevance and publication-date review. A successful
  77. script exit means the requests completed, not that the relevance targets passed.
  78. The reliability pilot probes readiness every 30 seconds for five minutes, then
  79. warms one query and sends seven requests at concurrency 1, 2, and 4. It checks
  80. that those requests use cached, consistent results. This measures a short health
  81. window and cached API handling, not multi-day uptime or uncached search capacity.
  82. The runner allows at most 12 benchmark cases and ten minutes per reliability run;
  83. it creates no schedule. Reports can contain queries and third-party result text.
  84. ---
  85. ## What it does
  86. AgentSearch wraps [SearXNG](https://github.com/searxng/searxng) with a FastAPI layer that adds the pieces agents usually need on top of raw search: deduplication, cross-engine scoring, content extraction, query expansion, domain trust scoring, prompt injection scrubbing, and failure-pattern analysis.
  87. **Standard stack** — `docker compose up` gives you search on `:3939`.
  88. **Private stack** — `docker compose -f docker-compose.yml -f examples/compose.private.yml up` adds an instance on `:3940` that routes search and content-fetch traffic through Tor with Snowflake transport. Encrypted DNS via CoreDNS → Cloudflare DoT. Network-level isolation — the private SearXNG instance runs on an internal-only Docker network with no route to the internet except through Tor.
  89. ## Search engines
  90. AgentSearch delegates engine support to the connected SearXNG instance. The authoritative list for a running stack is:
  91. ```bash
  92. curl "http://localhost:3939/engines"
  93. ```
  94. The bundled settings configure broad web, scholarly, news, reference, and developer sources using SearXNG 2026.9.5 (`searxng/searxng@sha256:55e1fa15a63ff04e79e213e6aa2837549877b0c6d60757cdb633ae9111cb5fea`). Both templates are checked against the pinned image in CI. Engine registration does not guarantee live availability: Google, Brave, Reuters, and other sources can still return blocks or rate limits. Inspect `/engines` and response diagnostics for the current instance. Existing runtime files need the [configuration upgrade steps](docs/upgrading.md#searxng-202695-configuration); changing the image alone does not migrate them.
  95. Run `./scripts/prepare-searxng.sh` to create ignored local runtime files at `searxng/settings.yml` and `searxng/settings.tor.yml` with generated SearXNG instance secrets. Do not commit those generated files.
  96. Because SearXNG is configured with `use_default_settings: true`, your live instance may expose additional enabled engines from the installed SearXNG catalog. Use the `engines=` query parameter to request specific engines, and use `/engines` to verify what is available in that deployment.
  97. Strategy modes can also use direct no-key providers for vertical search. These do not require paid search APIs: GitHub repository search, MDN search, Docker Hub search, PyPI package metadata, Wikipedia, Wikidata, Hacker News, arXiv, Crossref, OpenAlex, and Semantic Scholar are called directly when a mode selects them. SearXNG remains the broad-web provider for Google/Bing/Brave/DuckDuckGo-style engines, but Google, Startpage, Yahoo, and Reddit are best-effort explicit sources rather than defaults because they are commonly blocked or empty.
  98. ## Why not just use SearXNG directly?
  99. SearXNG finds pages. AgentSearch fetches and reads them, scores and deduplicates the results, caches them, scrubs prompt injections, detects paywalls, and falls back through several extraction strategies when the first one fails — in one API call. It also logs fetch outcomes so you can see which strategies and domains are actually working.
  100. You operate the containers, storage, and network connection. AgentSearch does not charge per query; hosting and maintenance still have costs. The evolver analyzes recorded failures and returns recommendations; it does not automatically modify the running system.
  101. ## Related projects
  102. For narrower tasks, these separate projects in the same GitHub account may be useful:
  103. - **[scholar](https://github.com/brcrusoe72/scholar-cli)**: scholarly search.
  104. - **[agent-read](https://github.com/brcrusoe72/agent-read)**: URL-to-text extraction.
  105. - **[agent-web](https://github.com/brcrusoe72/agent-web)**: browser interaction.
  106. See each repository for installation instructions and documentation.
  107. ## Endpoints
  108. ### Search
  109. | Endpoint | Method | What it does |
  110. |---|---|---|
  111. | `/search` | GET | Multi-engine web search with deduplication and scoring |
  112. | `/search/strategy` | GET | Named search modes: general, browser, code, academic, news, private, reference, community |
  113. | `/search/deep` | GET | Server-side query expansion — runs variations in parallel, fuses results |
  114. | `/search/extract` | GET | Search + inline content extraction in one call |
  115. | `/search/jobs` | GET | Job search across LinkedIn, Indeed, Glassdoor, ZipRecruiter |
  116. | `/search/policy` | GET | Policy and regulatory document search |
  117. | `/search/sources` | GET | Source discovery with institutional filtering |
  118. | `/search/sources/institutions` | GET | List source registry institutions |
  119. | `/search/stats` | GET | Query statistics and cache metrics |
  120. | `/news` | GET | Structured multi-source news with upstream diagnostics and explicit engine overrides |
  121. ### Content extraction
  122. | Endpoint | Method | What it does |
  123. |---|---|---|
  124. | `/read` | GET | Escalating extraction for any URL |
  125. | `/read/batch` | POST | Concurrent multi-URL extraction in one request |
  126. | `/providers/browser/fetch` | GET | Ephemeral browser render/extract for JS-rendered target pages |
  127. The extraction chain tries applicable strategies until one succeeds. PDF, YouTube, and site-specific handlers may run before the general fallbacks below:
  128. 1. Direct fetch + smart content selectors
  129. 2. Readability scoring (paragraph density vs link density)
  130. 3. User-agent rotation (Chrome/Safari/Firefox/Edge signatures)
  131. 4. Browser render/extract for JS-rendered target pages
  132. 5. Wayback Machine (CDX API → latest snapshot)
  133. 6. Google Cache
  134. 7. Search-about fallback (find coverage elsewhere)
  135. 8. Custom adapters (pluggable Python modules from disk)
  136. 9. PDF extraction (pdfplumber)
  137. 10. YouTube transcript (yt-dlp)
  138. The bundled extraction paths validate destinations and redirects, bound fetched content, and apply content scrubbing and paywall detection. Scrubbing is heuristic: returned text remains untrusted. In the Docker stack, browser rendering runs as a dedicated non-root service on an internal-only network, uses an ephemeral sandboxed context, and can reach targets only through the validating fetch proxy. It reports CAPTCHA/challenge pages instead of trying to bypass them. Custom adapters execute local Python code and must be reviewed before installation.
  139. ### Adaptation
  140. | Endpoint | Method | What it does |
  141. |---|---|---|
  142. | `/adapt/report` | POST | Report a fetch failure for a URL |
  143. | `/adapt/stats` | GET | View adaptation metrics and failure patterns |
  144. | `/adapt/evolve` | POST | Analyze failures and return recommendations (read-only) |
  145. ### Infrastructure
  146. | Endpoint | Method | What it does |
  147. |---|---|---|
  148. | `/health` | GET | Health check (API + SearXNG status) |
  149. | `/live` | GET | Cheap process liveness check; never calls upstream search |
  150. | `/ready` | GET | Bounded SearXNG readiness check; returns 503 when unusable |
  151. | `/engines` | GET | List available search engines and their status |
  152. | `/providers/health` | GET | Summarize provider health from recorded live attempts |
  153. | `/providers/stats` | GET | Rolling provider/SearXNG attempt telemetry |
  154. ## Quick examples
  155. ### Search with content extraction
  156. ```bash
  157. curl "http://localhost:3939/search/extract?q=python+async+patterns&count=3"
  158. ```
  159. Returns search results with extracted content inline — no second round-trip to `/read`.
  160. ### Strategy search
  161. ```bash
  162. curl "http://localhost:3939/search/strategy?q=fetch+api&mode=code&count=5"
  163. curl "http://localhost:3939/search?q=AI+regulation&mode=academic&count=5"
  164. curl "http://localhost:3939/search?q=Python&mode=reference&count=5"
  165. ```
  166. Modes validate or call only their declared sources instead of falling back silently: `general` queries DuckDuckGo, Brave, and Bing together, then tries browser discovery when results are insufficient, followed by direct reference/community providers if needed; `code` uses direct GitHub, MDN, Docker Hub, and PyPI providers; `academic` uses direct arXiv, Crossref, OpenAlex, and Semantic Scholar providers; `news` uses Reuters, Bing News, and DuckDuckGo News through SearXNG; `reference` uses direct Wikipedia and Wikidata providers; `community` uses direct Hacker News; `private` avoids broad general web engines.
  167. Ordinary `/search` and `mode=general` balance query-term coverage with engine agreement, source priors, and upstream position. Documentation queries prioritize matching page titles and relevant hostnames over homepages and search listings; this is a ranking signal, not proof that a site is official. Scores are ranking heuristics, not probabilities of correctness. Query matching improves some irrelevant-result failures but does not establish that a result answers the question. Upstream blocks remain visible even when another engine supplies results.
  168. Academic search tries a quoted title query for arXiv, then one explicit keyword
  169. query if the title lookup is empty, with a three-second pause between calls.
  170. An arXiv ID uses a direct identifier lookup. Crossref uses bibliographic queries.
  171. Ranking uses titles and authors, with small capped citation and provider-agreement
  172. signals. Provider call order does not give earlier sources a ranking advantage,
  173. and provider diversity cannot displace a more relevant paper. Matching DOI and
  174. arXiv identifiers merge duplicate records; identical titles alone do not.
  175. Author names remain visible in result snippets. Provider rate limits can still
  176. reduce coverage; inspect `meta.engine_attempts` before interpreting an empty result.
  177. In `mode=code`, explicit Python package or PyPI lookups preserve exact package
  178. matches ahead of provider diversity. Other code and academic results still need
  179. relevance review. Empty responses caused by upstream errors are not cached for
  180. the normal result TTL, so the next request can observe recovery.
  181. Browser discovery uses the existing sandboxed Chromium worker to read public
  182. DuckDuckGo search pages, then Bing if more results are needed. It needs no paid
  183. API, subscription, or API key. It is automatically available when
  184. `BROWSER_SERVICE_URL` is configured; set `BROWSER_DISCOVERY_ENABLED=0` on the API
  185. to disable it. Ordinary `/search` and `mode=general` invoke it when the initial
  186. web results are fewer than requested or SearXNG is unavailable. Explicit
  187. `engines=` requests and specialized/private modes keep their declared sources.
  188. Use `mode=browser` to search through the browser directly, independently of
  189. SearXNG:
  190. ```bash
  191. curl "http://localhost:3939/search?q=SQLite+write+ahead+logging+documentation&mode=browser&count=3"
  192. ```
  193. Each browser search visits at most two search pages (up to 20 results per page),
  194. with a bounded worker deadline and a shared browser concurrency limit. It
  195. extracts organic result cards, decodes tracking URLs, validates destinations,
  196. and applies domain exclusions before ranking. The browser keeps its sandbox,
  197. validating proxy and network isolation. Generic page rendering still blocks
  198. search-engine URLs; only the dedicated search path can visit its two fixed
  199. search endpoints. It does not log in or solve challenges. Blocks, empty pages,
  200. timeouts and temporary cooldowns remain visible in `meta.engine_attempts` under
  201. `browser_attempts`; SearXNG errors remain visible even after successful recovery.
  202. There are no vendor query credits, but external sites may still restrict traffic.
  203. The normal result cache prevents repeated successful searches; empty failures
  204. are not cached for the normal result TTL. This feature does not build a local
  205. web index or crawl every result page. Use `fetch=true` to extract returned pages.
  206. Each strategy response includes `meta.engine_attempts` with source/provider, query, raw result count, latency, and upstream errors so blocked or empty providers stay visible.
  207. Provider telemetry is available without running another probe:
  208. ```bash
  209. curl "http://localhost:3939/providers/health"
  210. curl "http://localhost:3939/providers/stats"
  211. ```
  212. Telemetry is in-memory and reflects live attempts since the API process started. It tracks attempts, successes, empty-result rate, errors, latency, last error, and last success per direct provider or SearXNG pack.
  213. ### Deep search (query expansion)
  214. ```bash
  215. curl "http://localhost:3939/search/deep?q=ethon+industrial+ai+platform&count=10"
  216. ```
  217. Server-side query variation + parallel execution + result fusion. It can find additional results, but query expansion does not guarantee better relevance.
  218. ### Read a URL (escalating extraction)
  219. ```bash
  220. curl "http://localhost:3939/read?url=https://example.com/paywalled-article"
  221. ```
  222. ```json
  223. {
  224. "url": "https://example.com/paywalled-article",
  225. "content": "Full article text extracted via Wayback Machine...",
  226. "strategy": "wayback",
  227. "chars": 4821,
  228. "cached": false,
  229. "strategies_tried": ["direct", "readability", "ua_rotation", "wayback"]
  230. }
  231. ```
  232. ### Browser render a JS page
  233. ```bash
  234. curl "http://localhost:3939/providers/browser/fetch?url=https://example.com/app&max_links=20"
  235. ```
  236. Returns rendered text, page title, final URL, extracted links, trust metadata, and `challenge_detected=true` if the page is a CAPTCHA or bot challenge.
  237. ### Batch read
  238. ```bash
  239. curl -X POST "http://localhost:3939/read/batch" \
  240. -H "Content-Type: application/json" \
  241. -d '{"urls": ["https://a.com", "https://b.com", "https://c.com"]}'
  242. ```
  243. ### Python SDK
  244. ```bash
  245. pip install agentsearch-client
  246. ```
  247. ```python
  248. from agentsearch import AgentSearch
  249. client = AgentSearch() # defaults to localhost:3939
  250. results = client.search("manufacturing OEE best practices", count=5)
  251. for r in results.results:
  252. print(f"{r.title} — {r.url}")
  253. # Content extraction
  254. page = client.read("https://example.com/article")
  255. print(page.content[:500])
  256. # Browser-rendered extraction for JS-heavy target pages
  257. rendered = client.browser_fetch("https://example.com/app", max_links=20)
  258. print(rendered.title, rendered.links[:3])
  259. # Batch read
  260. pages = client.read_batch(["https://a.com", "https://b.com"])
  261. print(f"{pages.successful}/{pages.total} succeeded")
  262. ```
  263. For authenticated instances, pass `token=...` or use `AGENT_SEARCH_TOKEN`,
  264. `AGENTSEARCH_TOKEN`, `~/.openclaw/workspace/credentials/agent-search-token.txt`,
  265. or `~/.config/agent-search/token`. CWD credential files are never loaded.
  266. Remote bearer-token connections require HTTPS unless the SDK caller explicitly
  267. sets `allow_insecure_http=True`.
  268. ### LangChain tool
  269. ```python
  270. from langchain.tools import tool
  271. import requests
  272. @tool
  273. def web_search(query: str) -> str:
  274. """Search the web using AgentSearch."""
  275. resp = requests.get("http://localhost:3939/search", params={"q": query, "count": 5})
  276. return "\n".join(
  277. f"- {r['title']}: {r['url']}\n {r['snippet']}"
  278. for r in resp.json()["results"]
  279. )
  280. ```
  281. ### MCP server (Claude Desktop, Cursor, Windsurf)
  282. ```bash
  283. python3 -m venv .venv-mcp
  284. .venv-mcp/bin/python -m pip install -e ./mcp-server
  285. .venv-mcp/bin/agent-search-mcp
  286. ```
  287. For authenticated instances, set `AGENT_SEARCH_TOKEN` or run:
  288. ```bash
  289. .venv-mcp/bin/agent-search-mcp --token "change-me"
  290. ```
  291. Add to Claude Desktop config:
  292. ```json
  293. {
  294. "mcpServers": {
  295. "agent-search": {
  296. "command": "/absolute/path/to/agent-search/.venv-mcp/bin/agent-search-mcp"
  297. }
  298. }
  299. }
  300. ```
  301. Install from the repository root. The separate environment applies the MCP package's declared dependency bounds without changing the API environment. Use the absolute executable path in your client config; on Windows it is `.venv-mcp\Scripts\agent-search-mcp.exe`. See [`mcp-server/README.md`](mcp-server/README.md) for details.
  302. ## Private stack (Tor + encrypted DNS)
  303. The optional private stack routes search and content-fetch traffic through Tor:
  304. ```
  305. ┌──────────┐ ┌──────────────┐ ┌───────────────┐ ┌──────────┐
  306. │ :3940 │───▶│ api-private │───▶│ searxng-priv │───▶│ Tor │──▶ Internet
  307. │ (agent) │ │ (FastAPI) │ │ (SearXNG) │ │(Snowflake│
  308. └──────────┘ └──────────────┘ └───────────────┘ │ + obfs4) │
  309. └──────────┘
  310. All containers use CoreDNS → Cloudflare DoT
  311. tor-internal network: no direct egress possible
  312. ```
  313. **What this gives you:**
  314. - The bundled configuration uses encrypted DNS and Tor with Snowflake transport. Network observers can still see connection metadata; this is not a guarantee of anonymity.
  315. - The private SearXNG instance lives on an internal-only Docker network with no internet route except through Tor.
  316. - Port 3939 = direct; port 3940 = Tor-routed and typically slower. Providers still receive your query text, and AgentSearch stores local query/fetch records and cached content.
  317. **Setup:**
  318. ```bash
  319. ./scripts/prepare-searxng.sh
  320. docker compose -f docker-compose.yml -f examples/compose.private.yml up -d --build
  321. ```
  322. All private stack configs live in [`examples/`](examples/) — copy and customize as needed.
  323. ## Architecture
  324. ```
  325. Port 3939 (direct) Port 3940 (Tor-anonymized)
  326. │ │
  327. ▼ ▼
  328. ┌─────────┐ ┌─────────────┐
  329. │ API │ │ api-private │
  330. │(FastAPI) │ │ (FastAPI) │
  331. ├─────────┤ ├─────────────┤
  332. │ dedup │ ┌─────────────────┐ │ same code │ ┌───────────────┐
  333. │ scoring │ │ SearXNG │ │ Tor egress │ │ SearXNG-priv │
  334. │ cache │──│ Google, Bing, │ │ only │──│ (tor-internal │
  335. │ scrub │ │ DDG, Brave, │ └─────────────┘ │ network) │
  336. │ killchn │ │ /engines list │ └───────┬───────┘
  337. │ trust │ └─────────────────┘ │
  338. │ evolver │ ┌─────┴─────┐
  339. └─────────┘ │ Tor │
  340. │ │ Snowflake │
  341. ▼ │ + obfs4 │
  342. ┌─────────┐ └───────────┘
  343. │ CoreDNS │──▶ Cloudflare DoT (encrypted DNS)
  344. └─────────┘
  345. ```
  346. ### Key modules
  347. | Module | What it does |
  348. |---|---|
  349. | `killchain.py` | Escalating content extraction and browser-render fallback |
  350. | `browser_renderer.py` | Ephemeral browser rendering, extraction, and challenge detection |
  351. | `main.py` | FastAPI endpoints, auth, and rate limiting |
  352. | `source_tracer.py` | Source provenance tracking and citation chains |
  353. | `scrubber.py` | Heuristic prompt injection detection and content sanitization |
  354. | `source_library.py` | Curated institutional source registry |
  355. | `domain_trust.py` | Heuristic domain trust scoring |
  356. | `evolver.py` | Failure analysis and read-only recommendations |
  357. | `content_cache.py` | URL-keyed content cache with TTL |
  358. | `query_expansion.py` | Server-side query variation and fusion |
  359. Plus: 5 pluggable adapters (Cloudflare bypass, Medium, 403 handler, parse error recovery, empty content fallback), MCP server, Python SDK, test suite.
  360. ## Integration case study
  361. The historical [research-agent walkthrough](case-studies/wolf.md) describes an authentication repair and an integration pattern. Its reported outcomes are specific to that system; the underlying agent and evaluation data are not included here. Treat it as an integration example, not a reproducible benchmark for the current release.
  362. ## Configuration
  363. Environment variables (set in `docker-compose.yml` or `.env`):
  364. | Variable | Default | Description |
  365. |---|---|---|
  366. | `SEARXNG_URL` | `http://searxng:8080` | SearXNG instance URL |
  367. | `SEARXNG_IMAGE` | pinned SearXNG digest (2026.9.5) | SearXNG container image; override only after a live engine comparison on the candidate |
  368. | `PYTHON_BASE_IMAGE` | pinned Python digest | API Docker base image; override only when intentionally upgrading |
  369. | `COREDNS_IMAGE` | pinned CoreDNS digest | Private-stack DNS image |
  370. | `SOCAT_IMAGE` | pinned socat digest | Private-stack TCP forwarder image |
  371. | `TOR_BASE_IMAGE` | pinned Debian digest | Private-stack Tor proxy base image |
  372. | `CACHE_TTL` | `3600` | Cache duration in seconds |
  373. | `RATE_LIMIT` | `60` | Max requests per minute |
  374. | `SQLITE_TIMEOUT` | `1.0` | SQLite lock wait timeout in seconds for query stats |
  375. | `FETCH_LOG_RETENTION_DAYS` | `30` | Delete fetch analytics rows older than this many days during SQLite maintenance |
  376. | `QUERY_LOG_RETENTION_DAYS` | `30` | Delete query analytics rows older than this many days during SQLite maintenance |
  377. | `SQLITE_MAINTENANCE_INTERVAL_SECONDS` | `3600` | Interval for expired content-cache cleanup, log retention, and `PRAGMA optimize` |
  378. | `SQLITE_VACUUM_MIN_DELETED_ROWS` | `1000` | Run `VACUUM` only when a maintenance pass deletes at least this many rows; set `0` to disable threshold vacuum |
  379. | `AGENT_SEARCH_TOKEN` | *(empty)* | Bearer token for auth (optional) |
  380. | `ADAPTERS_DIR` | `/app/adapters` | Path to pluggable adapter modules |
  381. ## Limitations and security notes
  382. - Search engines, news engines, rate limits, and failure modes depend on the connected SearXNG instance. `/engines` is the live source of truth.
  383. - Bearer auth is a simple local API gate, not a multi-user authorization system. Treat `AGENT_SEARCH_TOKEN` as a shared service token.
  384. - Rate limiting is in memory. It resets on restart and is per API process.
  385. - Query statistics use local SQLite with WAL and a bounded lock timeout. For high-volume multi-worker deployments, move query logging to an external database or telemetry backend.
  386. - The MCP package supports `mcp>=1.27.2,<2.3.0`. CI tests both 1.27.2 and 2.2.0 in isolated environments, including real stdio discovery, successful invocation, and error semantics. Upgrade deliberately and run the package/CI checks before publishing.
  387. - Content extraction validates the starting URL and every redirect hop before fetching redirected content, but fetched third-party pages are still untrusted and are scrubbed before being returned.
  388. - Google Cache is unreliable because public cache availability changes frequently.
  389. - The Tor/private stack is intentionally slower than direct search.
  390. ## Development
  391. ```bash
  392. pip install -r requirements.txt
  393. SEARXNG_URL=http://localhost:8080 uvicorn app.main:app --reload --port 3939
  394. # Run tests
  395. pytest tests/
  396. ```
  397. ## Release and GitHub governance
  398. - `Test` runs on every push and pull request.
  399. - `CodeQL` runs on push, pull request, and a weekly schedule.
  400. - Dependabot watches Python packages, Dockerfiles, and GitHub Actions.
  401. - Version releases are created from semantic tags such as `v2.0.1` or by manually running the `Release` workflow with a tag input.
  402. - Update [`CHANGELOG.md`](CHANGELOG.md) before creating a release tag.
  403. ## Contributing
  404. 1. Fork → branch → commit → PR.
  405. Issues and PRs welcome. If you're building an agent that needs search, this is for you.
  406. ## License
  407. The root AgentSearch API, SDK, Docker stack, and docs are MIT licensed. The MCP server under `mcp-server/` is AGPL-3.0 licensed; see [mcp-server/LICENSE](mcp-server/LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.