decodo-scraper
Search Google, scrape web pages, Amazon product pages, YouTube subtitles, or Reddit (post/subreddit) using the Decodo Scraper OpenClaw Skill.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/decodo--decodo-openclaw-skill)HTML
<a href="/listings/decodo--decodo-openclaw-skill"><img src="/api/listings/decodo--decodo-openclaw-skill/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
1. Clone this repo.
git clone https://github.com/Decodo/decodo-openclaw-skill.git
2. Install dependencies.
pip install -r requirements.txt
3. Set your Decodo auth token as an environment variable (or create a ```.env``` file in the project root):
# Linux/macOS Terminal export DECODO_AUTH_TOKEN="your_base64_token" # Windows (PowerShell) $env:DECODO_AUTH_TOKEN="your_base64_token"
# .env file DECODO_AUTH_TOKEN=your_base64_token
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 163 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: decodo-scraper
- description: Search Google, scrape web pages, Amazon product pages, YouTube subtitles, or Reddit (post/subreddit) using the Decodo Scraper OpenClaw Skill.
- homepage: https://decodo.com
- credentials:
- - DECODO_AUTH_TOKEN
- env:
- required:
- - DECODO_AUTH_TOKEN
- ---
- # Decodo Scraper OpenClaw Skill
- Use this skill to search Google, scrape any URL, or fetch YouTube subtitles via the [Decodo Web Scraping API](https://help.decodo.com/docs/web-scraping-api-google-search). **Search** outputs a JSON object of result sections; **Scrape URL** outputs plain markdown; **Amazon** and **Amazon search** output parsed product-page or search results (JSON). Amazon search uses `--query`. **YouTube subtitles** outputs transcript/subtitles. **Reddit post** and **Reddit subreddit** output post/listing content (JSON).
- **Authentication:** Set `DECODO_AUTH_TOKEN` (Basic auth token from Decodo Dashboard → Scraping APIs) in your environment or in a `.env` file in the repo root.
- **Errors:** On failure the script writes a JSON error to stderr and exits with code 1.
- ---
- ## Tools
- ### 1. Search Google
- Use this to find URLs, answers, or structured search results. The API returns a JSON object whose `results` key contains several sections (not all may be present for every query):
- | Section | Description |
- |--------|--------------|
- | `organic` | Main search results (titles, links, snippets). |
- | `ai_overviews` | AI-generated overviews or summaries when Google shows them. |
- | `paid` | Paid/sponsored results (ads). |
- | `related_questions` | “People also ask”–style questions and answers. |
- | `related_searches` | Suggested related search queries. |
- | `discussions_and_forums` | Forum or discussion results (e.g. Reddit, Stack Exchange). |
- The script outputs only the inner `results` object (these sections); pagination info (`page`, `last_visible_page`, `parse_status_code`) is not included.
- **Command:**
- ```bash
- python3 tools/scrape.py --target google_search --query "your search query"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target google_search --query "best laptops 2025"
- python3 tools/scrape.py --target google_search --query "python requests tutorial"
- ```
- Optional: `--geo US` or `--locale en` for location/language.
- ---
- ### 2. Scrape URL
- Use this to get the content of a specific web page. By default the API returns content as **Markdown** (cleaner for LLMs and lower token usage).
- **Command:**
- ```bash
- python3 tools/scrape.py --target universal --url "https://example.com"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target universal --url "https://example.com"
- python3 tools/scrape.py --target universal --url "https://news.ycombinator.com/"
- ```
- ---
- ### 3. Amazon product page
- Use this to get parsed data from an Amazon product (or other Amazon) page. Pass the product page URL as `--url`. The script sends `parse: true` and outputs the inner **results** object (e.g. `ads`, product details, etc.).
- **Command:**
- ```bash
- python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/PRODUCT_ID"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"
- ```
- ---
- ### 4. Amazon search
- Use this to search Amazon and get parsed results (search results list, delivery_postcode, etc.). Pass the search query as `--query`.
- **Command:**
- ```bash
- python3 tools/scrape.py --target amazon_search --query "your search query"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target amazon_search --query "laptop"
- ```
- ---
- ### 5. YouTube subtitles
- Use this to get subtitles/transcript for a YouTube video. Pass the **video ID** (e.g. from `youtube.com/watch?v=VIDEO_ID`) as `--query`.
- **Command:**
- ```bash
- python3 tools/scrape.py --target youtube_subtitles --query "VIDEO_ID"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"
- ```
- ---
- ### 6. Reddit post
- Use this to get the content of a Reddit post (thread). Pass the full post URL as `--url`.
- **Command:**
- ```bash
- python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/SUBREDDIT/comments/ID/..."
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/serious_next_day_thread_postgame_discussion/"
- ```
- ---
- ### 7. Reddit subreddit
- Use this to get the listing (posts) of a Reddit subreddit. Pass the subreddit URL as `--url`.
- **Command:**
- ```bash
- python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/SUBREDDIT/"
- ```
- **Examples:**
- ```bash
- python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"
- ```
- ---
- ## Summary
- | Action | Target | Argument | Example command |
- |--------------------|----------------------|------------|-----------------|
- | Search | `google_search` | `--query` | `python3 tools/scrape.py --target google_search --query "laptop"` |
- | Scrape page | `universal` | `--url` | `python3 tools/scrape.py --target universal --url "https://example.com"` |
- | Amazon product | `amazon` | `--url` | `python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"` |
- | Amazon search | `amazon_search` | `--query` | `python3 tools/scrape.py --target amazon_search --query "laptop"` |
- | YouTube subtitles | `youtube_subtitles` | `--query` | `python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"` |
- | Reddit post | `reddit_post` | `--url` | `python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/..."` |
- | Reddit subreddit | `reddit_subreddit` | `--url` | `python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"` |
- **Output:** Search → JSON (sections). Scrape URL → markdown. Amazon / Amazon search → JSON (results e.g. ads, product info, delivery_postcode). YouTube → transcript. Reddit → JSON (content).
README.md · README · 123 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # Decodo Scraper OpenClaw Skill
- 
- 
- 
- 
- <p align="center">
- <p align="center">
- <a href="https://dashboard.decodo.com/scrapers/pricing?utm_source=github&utm_medium=social&utm_campaign=openclaw"> <img src="https://github.com/user-attachments/assets/a1e52a9e-3da1-4081-b3c6-053aafb8f196"/></a>
- [](https://discord.gg/Ja8dqKgvbZ)
- ## Overview
- This [OpenClaw](https://openclaw.ai/) skill integrates [Decodo's Web Scraping API](https://decodo.com/scraping/web?utm_source=github&utm_medium=social&utm_campaign=openclaw) into any OpenClaw-compatible AI agent or LLM pipeline. It exposes these tools that agents can call directly:
- | Tool | Description | Perfect for |
- | :---------------- | :------------------------------------------------------------------- | :--------------------------------------------------------------------------------------- |
- | `google_search` | Real-time Google Search (SERP) results as structured JSON. | Market research, competitor analysis, news monitoring, fact-checking, RAG pipelines. |
- | `universal` | Scrape & parse any public webpage into clean Markdown. | Summarizing articles, content aggregation, building custom datasets, general web browsing for AI agents. |
- | `amazon` | Parse Amazon product page data (price, reviews, specs, ASIN). | eCommerce monitoring, price tracking, competitive intelligence, product research. |
- | `amazon_search` | Search Amazon for products by keyword and get parsed results. | Discovering products, tracking trends, and broad market analysis. |
- | `youtube_subtitles` | Extract subtitles/transcripts from YouTube videos (by video ID). | Video summarization, content analysis, sentiment tracking, accessibility. |
- | `reddit_post` | Fetch a Reddit post's content, comments, and metadata (by post URL). | Social listening, community sentiment analysis, trend tracking, and gathering user feedback. |
- | `reddit_subreddit` | Scrape Reddit subreddit listings (by subreddit URL). | Monitoring specific communities, content discovery, niche market research. |
- Backed by Decodo's residential and datacenter proxy infrastructure, the skill handles JavaScript rendering, bot detection bypass, and geo-targeting out of the box.
- ## Why use Decodo for your OpenClaw agent?
- * **Zero blocks & CAPTCHAs**. Backed by Decodo's proxy infrastructure from 125M+ locations, the skill automatically handles JavaScript rendering, bot detection, and CAPTCHA bypass.
- * **Real-time data**. Access fresh, up-to-the-minute web data directly within your AI agent's workflow.
- * **LLM-optimized output**. Data is returned in structured JSON or clean Markdown, making it easy for LLMs to understand and process.
- * **Scalability**. Designed for high-volume data collection, ensuring your agent can scale from small tasks to complex projects.
- * **Minimal Friction**. Easy setup with a single authentication token.
- ## Features
- - Real-time Google Search results scraping
- - Universal URL scraping
- - Amazon product page parsing (by URL)
- - Amazon search (by query)
- - YouTube subtitles/transcript by video ID
- - Reddit post content by URL
- - Reddit subreddit listing by URL
- - Structured JSON or Markdown results
- - Simple CLI interface compatible with any OpenClaw agent runtime
- - Designed for scalable AI agent web scraping
- - Minimal dependencies — just Python with Requests
- - Authentication via a single Base64 token from the [Decodo dashboard](https://dashboard.decodo.com/?utm_source=github&utm_medium=social&utm_campaign=openclaw)
- ## Prerequisites
- - [Python 3.9](https://www.python.org/downloads/) or higher
- - [Decodo account](https://dashboard.decodo.com/?utm_source=github&utm_medium=social&utm_campaign=openclaw) with access to the Web Scraping API
- - [OpenClaw](https://openclaw.ai/) installed on your machine
- ## Setup
- 1. Clone this repo.
- ```
- git clone https://github.com/Decodo/decodo-openclaw-skill.git
- ```
- 2. Install dependencies.
- ```
- pip install -r requirements.txt
- ```
- 3. Set your Decodo auth token as an environment variable (or create a ```.env``` file in the project root):
- ```
- # Linux/macOS Terminal
- export DECODO_AUTH_TOKEN="your_base64_token"
- # Windows (PowerShell)
- $env:DECODO_AUTH_TOKEN="your_base64_token"
- ```
- ```
- # .env file
- DECODO_AUTH_TOKEN=your_base64_token
- ```
- ## OpenClaw agent integration
- This skill ships with a [SKILL.md](https://github.com/Decodo/decodo-openclaw-skill/blob/main/SKILL.md) file that defines all tools in the OpenClaw skill format. OpenClaw-compatible agents automatically discover and invoke the tools from this file without additional configuration.
- To register the skill with your OpenClaw agent, point it at the repo root — the agent will read ```SKILL.md``` and expose ```google_search```, ```universal```, ```amazon```, ```amazon_search```, ```youtube_subtitles```, ```reddit_post```, and ```reddit_subreddit``` as callable tools.
- ## Usage
- ### Google Search
- Search Google and receive structured JSON. Results are grouped by type: **organic** (main results), **ai_overviews** (AI-generated summaries), **paid** (ads), **related_questions**, **related_searches**, **discussions_and_forums**, and others depending on the query.
- ```
- python3 tools/scrape.py --target google_search --query "your query"
- ```
- ### Scrape a URL
- Fetch and convert any webpage to a clean Markdown file:
- ```
- python3 tools/scrape.py --target universal --url "https://example.com/article"
- ```
- ### Amazon product page
- Fetch parsed data from an Amazon product page (e.g., ads, product details). Use the product URL:
- ```
- python3 tools/scrape.py --target amazon --url "https://www.amazon.com/dp/B09H74FXNW"
- ```
- ### Amazon search
- Search Amazon and get parsed results (e.g., results list, delivery_postcode):
- ```
- python3 tools/scrape.py --target amazon_search --query "laptop"
- ```
- ### YouTube subtitles
- Fetch subtitles/transcript for a YouTube video (use the video ID, e.g., from `?v=VIDEO_ID`):
- ```
- python3 tools/scrape.py --target youtube_subtitles --query "dFu9aKJoqGg"
- ```
- ### Reddit post
- Fetch a Reddit post’s content (use the full post URL):
- ```
- python3 tools/scrape.py --target reddit_post --url "https://www.reddit.com/r/nba/comments/17jrqc5/serious_next_day_thread_postgame_discussion/"
- ```
- ### Reddit subreddit
- Fetch a Reddit subreddit listing (use the subreddit URL):
- ```
- python3 tools/scrape.py --target reddit_subreddit --url "https://www.reddit.com/r/nba/"
- ```
- ## Related resources
- [Decodo Web Scraping API documentation](https://help.decodo.com/docs/web-scraping-api-introduction)
- [OpenClaw documentation](https://docs.openclaw.ai/start/getting-started)
- [ClaWHub – OpenClaw skill registry](https://docs.openclaw.ai/tools/clawhub)
- ## License
- All code is released under the [MIT License](https://github.com/Decodo/Decodo/blob/master/LICENSE).
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.