Skip to content
Back to search

decodo-scraper

SkillWorks with Claude CodeGitHub

Search Google, scrape web pages, Amazon product pages, YouTube subtitles, or Reddit (post/subreddit) using the Decodo Scraper OpenClaw Skill.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/decodo--decodo-openclaw-skill/badge.svg)](/listings/decodo--decodo-openclaw-skill)

HTML

<a href="/listings/decodo--decodo-openclaw-skill"><img src="/api/listings/decodo--decodo-openclaw-skill/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

1. Clone this repo.

git clone https://github.com/Decodo/decodo-openclaw-skill.git

2. Install dependencies.

pip install -r requirements.txt

3. Set your Decodo auth token as an environment variable (or create a ```.env``` file in the project root):

# Linux/macOS Terminal
export DECODO_AUTH_TOKEN="your_base64_token"

# Windows (PowerShell)
$env:DECODO_AUTH_TOKEN="your_base64_token"
# .env file
DECODO_AUTH_TOKEN=your_base64_token

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 163 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: decodo-scraper
  3. description: Search Google, scrape web pages, Amazon product pages, YouTube subtitles, or Reddit (post/subreddit) using the Decodo Scraper OpenClaw Skill.
  4. homepage: https://decodo.com
  5. credentials:
  6. - DECODO_AUTH_TOKEN
  7. env:
  8. required:
  9. - DECODO_AUTH_TOKEN
  10. ---
  11. # Decodo Scraper OpenClaw Skill
  12. Use this skill to search Google, scrape any URL, or fetch YouTube subtitles via the [Decodo Web Scraping API](https://help.decodo.com/docs/web-scraping-api-google-search). **Search** outputs a JSON object of result sections; **Scrape URL** outputs plain markdown; **Amazon** and **Amazon search** output parsed product-page or search results (JSON). Amazon search uses `--query`. **YouTube subtitles** outputs transcript/subtitles. **Reddit post** and **Reddit subreddit** output post/listing content (JSON).
  13. **Authentication:** Set `DECODO_AUTH_TOKEN` (Basic auth token from Decodo Dashboard → Scraping APIs) in your environment or in a `.env` file in the repo root.
  14. **Errors:** On failure the script writes a JSON error to stderr and exits with code 1.
  15. ---
  16. ## Tools
  17. ### 1. Search Google
  18. Use this to find URLs, answers, or structured search results. The API returns a JSON object whose `results` key contains several sections (not all may be present for every query):
  19. | Section | Description |
  20. |--------|--------------|
  21. | `organic` | Main search results (titles, links, snippets). |
  22. | `ai_overviews` | AI-generated overviews or summaries when Google shows them. |
  23. | `paid` | Paid/sponsored results (ads). |
  24. | `related_questions` | “People also ask”–style questions and answers. |
  25. | `related_searches` | Suggested related search queries. |
  26. | `discussions_and_forums` | Forum or discussion results (e.g. Reddit, Stack Exchange). |
  27. The script outputs only the inner `results` object (these sections); pagination info (`page`, `last_visible_page`, `parse_status_code`) is not included.
  28. **Command:**
  29. ```bash
  30. python3 tools/scrape.py --target google_search --query "your search query"
  31. ```
  32. **Examples:**
  33. ```bash
  34. python3 tools/scrape.py --target google_search --query "best laptops 2025"
  35. python3 tools/scrape.py --target google_search --query "python requests tutorial"
  36. ```
  37. Optional: `--geo US` or `--locale en` for location/language.
  38. ---
  39. ### 2. Scrape URL
  40. Use this to get the content of a specific web page. By default the API returns content as **Markdown** (cleaner for LLMs and lower token usage).
  41. **Command:**
  42. ```bash
  43. python3 tools/scrape.py --target universal --url "https://example.com"
  44. ```
  45. **Examples:**
  46. ```bash
  47. python3 tools/scrape.py --target universal --url "https://example.com"
  48. python3 tools/scrape.py --target universal --url "https://news.ycombinator.com/"
  49. ```
  50. ---
  51. ### 3. Amazon product page
  52. Use this to get parsed data from an Amazon product (or other Amazon) page. Pass the product page URL as `--url`. The script sends `parse: true` and outputs the inner **results** object (e.g. `ads`, product details, etc.).
  53. **Command:**
  54. ```bash
  55. python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/PRODUCT_ID"
  56. ```
  57. **Examples:**
  58. ```bash
  59. python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"
  60. ```
  61. ---
  62. ### 4. Amazon search
  63. Use this to search Amazon and get parsed results (search results list, delivery_postcode, etc.). Pass the search query as `--query`.
  64. **Command:**
  65. ```bash
  66. python3 tools/scrape.py --target amazon_search --query "your search query"
  67. ```
  68. **Examples:**
  69. ```bash
  70. python3 tools/scrape.py --target amazon_search --query "laptop"
  71. ```
  72. ---
  73. ### 5. YouTube subtitles
  74. Use this to get subtitles/transcript for a YouTube video. Pass the **video ID** (e.g. from `youtube.com/watch?v=VIDEO_ID`) as `--query`.
  75. **Command:**
  76. ```bash
  77. python3 tools/scrape.py --target youtube_subtitles --query "VIDEO_ID"
  78. ```
  79. **Examples:**
  80. ```bash
  81. python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"
  82. ```
  83. ---
  84. ### 6. Reddit post
  85. Use this to get the content of a Reddit post (thread). Pass the full post URL as `--url`.
  86. **Command:**
  87. ```bash
  88. python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/SUBREDDIT/comments/ID/..."
  89. ```
  90. **Examples:**
  91. ```bash
  92. python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/serious_next_day_thread_postgame_discussion/"
  93. ```
  94. ---
  95. ### 7. Reddit subreddit
  96. Use this to get the listing (posts) of a Reddit subreddit. Pass the subreddit URL as `--url`.
  97. **Command:**
  98. ```bash
  99. python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/SUBREDDIT/"
  100. ```
  101. **Examples:**
  102. ```bash
  103. python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"
  104. ```
  105. ---
  106. ## Summary
  107. | Action | Target | Argument | Example command |
  108. |--------------------|----------------------|------------|-----------------|
  109. | Search | `google_search` | `--query` | `python3 tools/scrape.py --target google_search --query "laptop"` |
  110. | Scrape page | `universal` | `--url` | `python3 tools/scrape.py --target universal --url "https://example.com"` |
  111. | Amazon product | `amazon` | `--url` | `python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"` |
  112. | Amazon search | `amazon_search` | `--query` | `python3 tools/scrape.py --target amazon_search --query "laptop"` |
  113. | YouTube subtitles | `youtube_subtitles` | `--query` | `python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"` |
  114. | Reddit post | `reddit_post` | `--url` | `python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/..."` |
  115. | Reddit subreddit | `reddit_subreddit` | `--url` | `python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"` |
  116. **Output:** Search → JSON (sections). Scrape URL → markdown. Amazon / Amazon search → JSON (results e.g. ads, product info, delivery_postcode). YouTube → transcript. Reddit → JSON (content).
README.md · README · 123 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Decodo Scraper OpenClaw Skill
  2. ![Python Version](https://img.shields.io/badge/python-3.9%2B-blue.svg?logo=python)
  3. ![License](https://img.shields.io/github/license/decodo/decodo)
  4. ![powered by Decodo](https://img.shields.io/badge/powered_by-Decodo-red?)
  5. ![GitHub Repo stars](https://img.shields.io/github/stars/Decodo/decodo-openclaw-skill)
  6. <p align="center">
  7. <p align="center">
  8. <a href="https://dashboard.decodo.com/scrapers/pricing?utm_source=github&utm_medium=social&utm_campaign=openclaw"> <img src="https://github.com/user-attachments/assets/a1e52a9e-3da1-4081-b3c6-053aafb8f196"/></a>
  9. [![](https://dcbadge.limes.pink/api/server/https://discord.gg/Ja8dqKgvbZ)](https://discord.gg/Ja8dqKgvbZ)
  10. ## Overview
  11. This [OpenClaw](https://openclaw.ai/) skill integrates [Decodo's Web Scraping API](https://decodo.com/scraping/web?utm_source=github&utm_medium=social&utm_campaign=openclaw) into any OpenClaw-compatible AI agent or LLM pipeline. It exposes these tools that agents can call directly:
  12. | Tool | Description | Perfect for |
  13. | :---------------- | :------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- |
  14. | `google_search` | Real-time Google Search (SERP) results as structured JSON. | Market research, competitor analysis, news monitoring, fact-checking, RAG pipelines. |
  15. | `universal` | Scrape & parse any public webpage into clean Markdown. | Summarizing articles, content aggregation, building custom datasets, general web browsing for AI agents. |
  16. | `amazon` | Parse Amazon product page data (price, reviews, specs, ASIN). | eCommerce monitoring, price tracking, competitive intelligence, product research. |
  17. | `amazon_search` | Search Amazon for products by keyword and get parsed results. | Discovering products, tracking trends, and broad market analysis. |
  18. | `youtube_subtitles` | Extract subtitles/transcripts from YouTube videos (by video ID). | Video summarization, content analysis, sentiment tracking, accessibility. |
  19. | `reddit_post` | Fetch a Reddit post's content, comments, and metadata (by post URL). | Social listening, community sentiment analysis, trend tracking, and gathering user feedback. |
  20. | `reddit_subreddit` | Scrape Reddit subreddit listings (by subreddit URL). | Monitoring specific communities, content discovery, niche market research. |
  21. Backed by Decodo's residential and datacenter proxy infrastructure, the skill handles JavaScript rendering, bot detection bypass, and geo-targeting out of the box.
  22. ## Why use Decodo for your OpenClaw agent?
  23. * **Zero blocks & CAPTCHAs**. Backed by Decodo's proxy infrastructure from 125M+ locations, the skill automatically handles JavaScript rendering, bot detection, and CAPTCHA bypass.
  24. * **Real-time data**. Access fresh, up-to-the-minute web data directly within your AI agent's workflow.
  25. * **LLM-optimized output**. Data is returned in structured JSON or clean Markdown, making it easy for LLMs to understand and process.
  26. * **Scalability**. Designed for high-volume data collection, ensuring your agent can scale from small tasks to complex projects.
  27. * **Minimal Friction**. Easy setup with a single authentication token.
  28. ## Features
  29. - Real-time Google Search results scraping
  30. - Universal URL scraping
  31. - Amazon product page parsing (by URL)
  32. - Amazon search (by query)
  33. - YouTube subtitles/transcript by video ID
  34. - Reddit post content by URL
  35. - Reddit subreddit listing by URL
  36. - Structured JSON or Markdown results
  37. - Simple CLI interface compatible with any OpenClaw agent runtime
  38. - Designed for scalable AI agent web scraping
  39. - Minimal dependencies — just Python with Requests
  40. - Authentication via a single Base64 token from the [Decodo dashboard](https://dashboard.decodo.com/?utm_source=github&utm_medium=social&utm_campaign=openclaw)
  41. ## Prerequisites
  42. - [Python 3.9](https://www.python.org/downloads/) or higher
  43. - [Decodo account](https://dashboard.decodo.com/?utm_source=github&utm_medium=social&utm_campaign=openclaw) with access to the Web Scraping API
  44. - [OpenClaw](https://openclaw.ai/) installed on your machine
  45. ## Setup
  46. 1. Clone this repo.
  47. ```
  48. git clone https://github.com/Decodo/decodo-openclaw-skill.git
  49. ```
  50. 2. Install dependencies.
  51. ```
  52. pip install -r requirements.txt
  53. ```
  54. 3. Set your Decodo auth token as an environment variable (or create a ```.env``` file in the project root):
  55. ```
  56. # Linux/macOS Terminal
  57. export DECODO_AUTH_TOKEN="your_base64_token"
  58. # Windows (PowerShell)
  59. $env:DECODO_AUTH_TOKEN="your_base64_token"
  60. ```
  61. ```
  62. # .env file
  63. DECODO_AUTH_TOKEN=your_base64_token
  64. ```
  65. ## OpenClaw agent integration
  66. This skill ships with a [SKILL.md](https://github.com/Decodo/decodo-openclaw-skill/blob/main/SKILL.md) file that defines all tools in the OpenClaw skill format. OpenClaw-compatible agents automatically discover and invoke the tools from this file without additional configuration.
  67. To register the skill with your OpenClaw agent, point it at the repo root — the agent will read ```SKILL.md``` and expose ```google_search```, ```universal```, ```amazon```, ```amazon_search```, ```youtube_subtitles```, ```reddit_post```, and ```reddit_subreddit``` as callable tools.
  68. ## Usage
  69. ### Google Search
  70. Search Google and receive structured JSON. Results are grouped by type: **organic** (main results), **ai_overviews** (AI-generated summaries), **paid** (ads), **related_questions**, **related_searches**, **discussions_and_forums**, and others depending on the query.
  71. ```
  72. python3 tools/scrape.py --target google_search --query "your query"
  73. ```
  74. ### Scrape a URL
  75. Fetch and convert any webpage to a clean Markdown file:
  76. ```
  77. python3 tools/scrape.py --target universal --url "https://example.com/article"
  78. ```
  79. ### Amazon product page
  80. Fetch parsed data from an Amazon product page (e.g., ads, product details). Use the product URL:
  81. ```
  82. python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"
  83. ```
  84. ### Amazon search
  85. Search Amazon and get parsed results (e.g., results list, delivery_postcode):
  86. ```
  87. python3 tools/scrape.py --target amazon_search --query "laptop"
  88. ```
  89. ### YouTube subtitles
  90. Fetch subtitles/transcript for a YouTube video (use the video ID, e.g., from `?v=VIDEO_ID`):
  91. ```
  92. python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"
  93. ```
  94. ### Reddit post
  95. Fetch a Reddit post’s content (use the full post URL):
  96. ```
  97. python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/serious_next_day_thread_postgame_discussion/"
  98. ```
  99. ### Reddit subreddit
  100. Fetch a Reddit subreddit listing (use the subreddit URL):
  101. ```
  102. python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"
  103. ```
  104. ## Related resources
  105. [Decodo Web Scraping API documentation](https://help.decodo.com/docs/web-scraping-api-introduction)
  106. [OpenClaw documentation](https://docs.openclaw.ai/start/getting-started)
  107. [ClaWHub – OpenClaw skill registry](https://docs.openclaw.ai/tools/clawhub)
  108. ## License
  109. All code is released under the [MIT License](https://github.com/Decodo/Decodo/blob/master/LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.