Skip to content
Back to search

webfetch-sdk

Official clients for the webfetch.io scraping and search API, in JavaScript, Python and PHP. MIT licensed.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/tuxxin--webfetch-sdk/badge.svg)](/listings/tuxxin--webfetch-sdk)

HTML

<a href="/listings/tuxxin--webfetch-sdk"><img src="/api/listings/tuxxin--webfetch-sdk/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 119 lines
  1. # Webfetch SDKs
  2. Official clients for the [webfetch.io](https://webfetch.io) scraping and search
  3. API, in JavaScript, Python and PHP. MIT licensed.
  4. Each one wraps `GET https://api.webfetch.io/v1/` and the search endpoints
  5. (`/v1/google`, `/v1/bing`, `/v1/yahoo`, `/v1/brave`). The API is ScrapingBee-compatible, so
  6. parameter names, defaults and credit costs match.
  7. | Language | Install | Requires |
  8. |---|---|---|
  9. | JavaScript | `npm install @tuxxin/webfetch` | Node 18+, no runtime dependencies |
  10. | Python | `pip install webfetch-io` (imports as `webfetch`) | Python 3.9+, `httpx` |
  11. | PHP | `composer require tuxxin/webfetch` | PHP 8.1+, ext-curl |
  12. ## The shape is the same in all three
  13. ```
  14. client = Webfetch(api_key)
  15. result = client.fetch(url, render_js=True, ...)
  16. result.content the page, as bytes or a string
  17. result.credits what this call cost, from the Wf-Cost header
  18. result.status the status the target returned
  19. result.resolved_url where it ended up after redirects
  20. ```
  21. ## Search
  22. ```
  23. results = client.bing("python asyncio tutorial", country_code="us")
  24. results["organic_results"] [{position, title, url, displayed_url, description, ...}]
  25. results["top_ads"] [{position, title, url, tracking_url, domain, ...}]
  26. client.search("yahoo", query, search_type="ads") the ads only, same price
  27. ```
  28. The response has the shape of ScrapingBee's Google Search API.
  29. Needs a paid plan with the robots.txt opt-out turned on for the key in the
  30. [key portal](https://webfetch.io/key): every engine's robots.txt disallows its
  31. result pages. Google, Bing, Yahoo and Brave are all open. 10 credits a page, 15 with `light_request` false; a failed
  32. search costs nothing. Reference: <https://webfetch.io/documentation#search>.
  33. ## Credits
  34. | Request | Credits |
  35. |---|---|
  36. | No JavaScript rendering | 1 |
  37. | JavaScript rendering (the default) | 5 |
  38. | Premium proxy | 10, or 25 with rendering |
  39. | Stealth proxy | 75 |
  40. | Search, light request (the default) | 10 |
  41. | Search, `light_request` false | 15 |
  42. | Any failed request | 0 |
  43. `mode=auto` tries those routes cheapest first and charges only the one that
  44. worked; `max_cost` caps it. An anti-bot challenge or CAPTCHA page is never
  45. charged.
  46. `render_js` defaults to true, so a call that does not set it costs 5 credits.
  47. Pass `render_js=false` for a plain HTTP fetch at 1 credit.
  48. ## A key is required
  49. There is no keyless tier. A [free account](https://webfetch.io/signup) starts with
  50. 1,000 credits and gets 250 more each month (banking up to 1,000), and needs only a
  51. confirmed email address. Each client
  52. raises `WebfetchError` immediately when constructed without a key, rather than
  53. spending a round trip on a request that cannot succeed.
  54. ## Errors
  55. Every SDK raises the same three-level hierarchy, so you can catch broadly or
  56. narrowly:
  57. ```
  58. WebfetchError anything went wrong
  59. WebfetchApiError the API answered with an error status; carries .status and .code
  60. WebfetchRateLimitError 429, or 503 when free capacity is busy; carries .retry_after
  61. ```
  62. A failed call is never billed, so you do not need to reconcile credits after an
  63. error.
  64. ## MCP server
  65. Webfetch is also a remote [MCP](https://modelcontextprotocol.io) server, so an AI
  66. assistant can fetch pages and run searches as tools, with no SDK in between.
  67. | | |
  68. |---|---|
  69. | Endpoint | `https://webfetch.io/mcp` (Streamable HTTP, POST only) |
  70. | Registry | `io.webfetch/webfetch` in the [official MCP Registry](https://registry.modelcontextprotocol.io); manifest: [`server.json`](server.json) |
  71. | Auth | `Authorization: Bearer wfk_...` on every request. A [free key](https://webfetch.io/signup) starts with 1,000 credits and gets 250 more each month |
  72. | Docs | <https://webfetch.io/documentation#mcp> |
  73. | Tool | What it does |
  74. |---|---|
  75. | `fetch_page` | A URL as markdown, plain text or raw HTML. JavaScript is rendered by default (5 credits); `render_js` false is a plain fetch (1 credit) |
  76. | `extract_data` | JSON built from CSS or XPath selectors, so the model gets fields instead of a whole page |
  77. | `search_web` | One page of Google, Bing, Yahoo or Brave results as JSON: organic results, ads, related searches and questions. 10 credits a page; needs search access on the key |
  78. A failed call costs nothing. Claude Code, in `.mcp.json`:
  79. ```json
  80. {
  81. "mcpServers": {
  82. "webfetch": {
  83. "type": "http",
  84. "url": "https://webfetch.io/mcp",
  85. "headers": { "Authorization": "Bearer ${WEBFETCH_KEY}" }
  86. }
  87. }
  88. }
  89. ```
  90. ## Documentation
  91. <https://webfetch.io/documentation>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.