Skip to content
Back to search

bulkgrid

MarketplaceWorks with Claude CodeGitHub

Bulkgrid - Better AI Answers From Sources You Trust

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked yesterday
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/bulkgrid--bulkgrid/badge.svg)](/listings/bulkgrid--bulkgrid)

HTML

<a href="/listings/bulkgrid--bulkgrid"><img src="/api/listings/bulkgrid--bulkgrid/badge.svg" alt="findsafeskills safety rating" /></a>

Install

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

.claude-plugin/marketplace.json · Manifest · 12 lines
  1. {
  2. "name": "bulkgrid",
  3. "owner": { "name": "Bulkgrid AB" },
  4. "plugins": [
  5. {
  6. "name": "bulkgrid",
  7. "source": "./plugins/bulkgrid-claude",
  8. "description": "Search indexed Bulkgrid sources and read documents with citations.",
  9. "license": "Apache-2.0"
  10. }
  11. ]
  12. }
README.md · README · 237 lines
  1. <p align="center">
  2. <a href="https://bulkgrid.com">
  3. <img src="docs/logo/bgicon.png" alt="Bulkgrid logo" width="112" height="112" />
  4. </a>
  5. </p>
  6. <h1 align="center">Bulkgrid</h1>
  7. <p align="center"><strong>Search the web. Map websites. Crawl content. Build your index.</strong></p>
  8. <p align="center">
  9. Web data and trusted knowledge for your applications and AI agents.
  10. </p>
  11. <p align="center">
  12. <a href="https://bulkgrid.com">Website</a> ·
  13. <a href="https://bulkgrid.com/docs">Documentation</a> ·
  14. <a href="https://bulkgrid.com/dashboard">Dashboard</a> ·
  15. <a href="https://www.npmjs.com/package/@bulkgrid/sdk">TypeScript</a> ·
  16. <a href="packages/sdk-python">Python</a>
  17. </p>
  18. ## One platform, from discovery to retrieval
  19. Bulkgrid turns web pages and repository content into useful data. Discover information with live search, map a website's URLs, collect page content, extract structured fields, and maintain a searchable index for your apps and agents.
  20. | Capability | What you can do | TypeScript SDK |
  21. | ------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------- |
  22. | **[Search](docs/search.mdx)** | Find web pages, images, or news. Optionally crawl their source pages in the same run. | `client.search()` |
  23. | **[Map](docs/map.mdx)** | Discover URLs from a website's sitemaps, with keyword filtering and a result limit. | `client.map()` |
  24. | **[Crawl](docs/crawl.mdx)** | Fetch specific URLs as Markdown, HTML, text, or screenshots. | `client.crawl()` |
  25. | **[Deep Crawl](docs/deep-crawl.mdx)** | Follow links within page, depth, and path boundaries. | `client.deepCrawl()` |
  26. | **[Extract](docs/extraction.mdx)** | Turn page content into structured data using a query and JSON schema. | `client.extract()` |
  27. | **[Index](docs/index.mdx)** | Search indexed website and GitHub content, curate collections, and refresh sources. | `client.index.search()` |
  28. Use these capabilities independently or combine them: **Search or Map → Crawl or Extract**, or **Sources → Index → Collections → Retrieval**. Map returns sitemap URLs; it does not fetch page content. Live search returns ranked results immediately. Map, Crawl, Deep Crawl, and Extract return runs that you can inspect for progress and results.
  29. ## Get started
  30. Create a workspace in the [dashboard](https://bulkgrid.com/dashboard), then create an API key in [Settings → API Keys](https://bulkgrid.com/dashboard/settings/api-keys). Keep the key in your backend environment.
  31. ### TypeScript / JavaScript
  32. Requires Node.js 22 or newer.
  33. ```bash
  34. npm install @bulkgrid/sdk
  35. export BULKGRID_API_KEY='bg_live_your_api_key'
  36. ```
  37. Search the live web:
  38. ```js
  39. import { BulkgridClient } from "@bulkgrid/sdk";
  40. const client = new BulkgridClient({
  41. apiKey: process.env.BULKGRID_API_KEY ?? "",
  42. });
  43. const search = await client.search({
  44. query: "open source web crawlers",
  45. limit: 5,
  46. });
  47. console.log(search.results);
  48. console.log(search.run.id); // Reopen saved results with client.runs.get(id)
  49. ```
  50. Set `type: 'images'` or `type: 'news'` for other search modes. Add `crawl: true` to collect the results' source pages in the same run.
  51. Map a website or crawl a known page:
  52. ```js
  53. const map = await client.map({
  54. url: "https://example.com",
  55. config: { search: "docs", limit: 100 },
  56. });
  57. const crawl = await client.crawl({
  58. urls: ["https://example.com"],
  59. options: { formats: ["markdown"] },
  60. });
  61. console.log(map.id, crawl.id);
  62. ```
  63. Use `client.runs.get(id)` to check progress. A completed map exposes its URLs in `result_data.links`; crawled pages are available through `client.runs.results(id)` and `client.runs.getResultContent(...)`. See [Runs and results](docs/runs-and-results.mdx) for polling, pagination, retry, and cancellation.
  64. Search your index:
  65. ```js
  66. const evidence = await client.index.search({
  67. query: "How does authentication work?",
  68. limit: 5,
  69. });
  70. for (const result of evidence.results) {
  71. console.log(result.text_content, result.url);
  72. }
  73. ```
  74. Index retrieval requires indexed content in collections accessible to your key. Manage that content with `client.index.sources` and `client.index.collections`. Public website and GitHub sources can be reused across workspaces; private sources stay within their workspace. See [Sources](docs/sources.mdx) and [Collections](docs/collections.mdx).
  75. Search requires `search:query`. Map and Crawl require `runs:write`; inspecting runs requires `runs:read`, and fetching page results requires `results:read`. See [API-key scopes](docs/api-key-scopes.mdx) for extraction and source-management permissions, and the [SDK guide](docs/sdk.mdx) for full examples.
  76. ### Python
  77. The Python SDK includes synchronous and asynchronous clients for Python 3.10+.
  78. ```bash
  79. pip install bulkgrid
  80. ```
  81. ```python
  82. import os
  83. from bulkgrid import BulkgridClient
  84. with BulkgridClient(api_key=os.environ['BULKGRID_API_KEY']) as client:
  85. search = client.search({'query': 'open source web crawlers', 'limit': 5})
  86. print(search['results'])
  87. evidence = client.index.search({'query': 'authentication setup', 'limit': 5})
  88. print(evidence['results'])
  89. ```
  90. Python uses methods such as `deep_crawl()` and `get_result_content()` while preserving API field names in dictionaries. See the [Python SDK README](packages/sdk-python/README.md) for async usage, configuration, and local installation. Repository changes may appear before their corresponding npm or PyPI release.
  91. ### Upgrading from an earlier SDK
  92. The 0.4 API separates live discovery from indexed retrieval:
  93. | Earlier interface | Current interface |
  94. | --------------------------------------------- | --------------------------------------------------------- |
  95. | `client.search()` for indexed content | `client.index.search()` |
  96. | `client.sources` / `client.collections` | `client.index.sources` / `client.index.collections` |
  97. | `/api/v1/sources/*` / `/api/v1/collections/*` | `/api/v1/index/sources/*` / `/api/v1/index/collections/*` |
  98. | `/api/v1/search` for indexed content | `/api/v1/index/search` |
  99. **`client.search()` and `/api/v1/search` now perform live web search.** Earlier source and collection routes have no compatibility aliases. See the [SDK changelog](packages/sdk/CHANGELOG.md) before upgrading.
  100. ## Connect your AI agent
  101. Add the remote MCP server in your client's settings:
  102. ```text
  103. https://bulkgrid.com/api/v1/mcp
  104. ```
  105. Complete browser sign-in, choose your workspace, and select the collections the connection can access. MCP retrieval searches indexed content and returns passages with source URLs.
  106. Claude and ChatGPT use the same MCP service. The updated integration offers these capabilities when the supporting service version is deployed:
  107. | App capability | Required consent |
  108. | --- | --- |
  109. | Search indexed content, list sources and collections, read documents | Default retrieval access |
  110. | Crawl, Deep Crawl, Map, Extract, and read run results | Optional **Crawl, map and extract** |
  111. | Add or refresh index sources and manage collection membership | Optional **Manage the index** |
  112. Both optional choices are off by default. Crawling and extraction may consume credits, and sources may refresh automatically. Index management also grants run permissions used by Crawl and Map. Monitoring remains available through scoped API keys, outside OAuth app access.
  113. For clients using `mcpServers` configuration:
  114. ```json
  115. {
  116. "mcpServers": {
  117. "bulkgrid": {
  118. "url": "https://bulkgrid.com/api/v1/mcp"
  119. }
  120. }
  121. }
  122. ```
  123. Try: “Use Bulkgrid to find how authentication works in our documentation. Search the relevant collection and cite the source URLs.”
  124. For an existing connection, revoke its access in **Settings → Connected Apps**, then reconnect and approve the desired optional capabilities. Updating the plugin or refreshing a token does not add permissions. See [MCP setup](docs/mcp-server.mdx) for instructions and [agent plugins](AGENT_PLUGINS.md) for deployment and client acceptance status.
  125. ## CLI and agent skills
  126. The skills target CLI 0.6.0 or newer for web and index search, crawling, extraction, document parsing, and source management. Browser login is the default; an API key is optional and takes precedence when `BULKGRID_API_KEY` is set.
  127. ```bash
  128. npx @bulkgrid/cli --version
  129. npx @bulkgrid/cli login
  130. npx @bulkgrid/cli search 'solar panel recycling' --limit 5
  131. npx @bulkgrid/cli crawl https://example.com --options '{"formats":["markdown"]}'
  132. npx @bulkgrid/cli parse ./report.pdf
  133. npx @bulkgrid/cli index search 'authentication setup' --limit 5
  134. npx @bulkgrid/cli sources find 'example.com'
  135. ```
  136. Approve **Crawl, map, extract and parse** for web operations and document parsing, and **Manage the index** for source management. Use `status` to inspect the active login. Existing grants need logout/login to approve added scopes on the same deployment.
  137. The prepared `setup skills` installer copies all six bundled skills into the project's `.agents/skills` without authentication or network access. Use `--global` or `--directory` for another destination. Existing edits are preserved unless you pass `--force`; `--dry-run` previews changes. This checkout's CLI implementation is still 0.5.0 and must be synced to 0.6.0 before releasing these updated instructions. See the [CLI README](packages/cli/README.md) for the checkout's implementation and installer details.
  138. | Skill | Purpose |
  139. | ---------------------------------------------------- | ------------------------------------------------------------------ |
  140. | [bulkgrid-index](skills/bulkgrid-index/SKILL.md) | Search indexed sources and collections, read document passages, and cite evidence. |
  141. | [bulkgrid-search](skills/bulkgrid-search/SKILL.md) | Search the web for pages, images, and news. |
  142. | [bulkgrid-crawl](skills/bulkgrid-crawl/SKILL.md) | Crawl websites, map URLs, collect bounded content, and retrieve outputs. |
  143. | [bulkgrid-extract](skills/bulkgrid-extract/SKILL.md) | Parse web pages into structured fields, design schemas, and validate extraction results. |
  144. | [bulkgrid-parse](skills/bulkgrid-parse/SKILL.md) | Convert local documents and images into Markdown and retrieve parse results. |
  145. | [bulkgrid-sources](skills/bulkgrid-sources/SKILL.md) | Inspect sources, manage ingestion, and organize collection access. |
  146. All six skills are bundled in the CLI package and both plugin bundles, generated from the same sources. They prefer CLI commands when a terminal is available and retain MCP/SDK fallback guidance. Skills do not grant permissions. `bulkgrid search` searches the live web; `bulkgrid index search` and MCP `search` retrieve indexed passages.
  147. For **parsing**, use `bulkgrid-parse` with CLI 0.6.0 for local documents and images, Crawl for readable web pages, and Extract for schema-based JSON from URLs. See the [skill coverage review](AGENT_PLUGINS.md#skill-coverage) for workflows and capability boundaries.
  148. ## In this repository
  149. | Component | Location |
  150. | ------------------------ | -------------------------------------------------------------- |
  151. | TypeScript SDK | [packages/sdk](packages/sdk) |
  152. | Python SDK | [packages/sdk-python](packages/sdk-python) |
  153. | CLI and MCP client setup | [packages/cli](packages/cli) |
  154. | Agent plugins | [AGENT_PLUGINS.md](AGENT_PLUGINS.md) |
  155. | Shared agent skills | [skills](skills) |
  156. | Documentation | [docs](docs) |
  157. | OpenAPI specification | [openapi/bulkgrid.openapi.json](openapi/bulkgrid.openapi.json) |
  158. | Runnable examples | [examples](examples) |
  159. These tools connect to the hosted Bulkgrid service. The API backend, MCP service, crawling and indexing engines, database, and billing are maintained in the private platform repository. This repository contains the public developer tools, not a self-hostable backend.
  160. ## Development and contributions
  161. ```bash
  162. npm ci
  163. npm run check
  164. node packages/cli/dist/cli.js --help
  165. ```
  166. Use Node.js 22+ and Python 3.10+ to validate all packages. See [CONTRIBUTING.md](CONTRIBUTING.md) for checks and [MAINTAINERS.md](MAINTAINERS.md) for the platform export workflow. API contracts and SDK types follow the platform schemas; plugins, shared skills, and public tooling are maintained here.
  167. ## License
  168. Agent plugin resources and shared skills use [Apache-2.0](packages/agent-core/LICENSE). Bulkgrid branding retains its trademark rights. SDK and CLI code remains `UNLICENSED`; the Python SDK declares a proprietary license. The documentation template's existing license is preserved in [docs/LICENSE](docs/LICENSE).
  169. [Documentation](https://bulkgrid.com/docs) · [Dashboard](https://bulkgrid.com/dashboard) · [X](https://x.com/bulkgrid) · [Support](mailto:support@bulkgrid.com)

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.