web-scraping
This skill activates for web scraping and Actor development. It proactively discovers APIs via traffic interception, recommends optimal strategy (traffic interception/sitemap/API/DOM scraping/hybrid), and implements iteratively. For production, it guides TypeScript Actor creation via Apify CLI.
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Badge for your README
Markdown
[](/listings/yfe404--web-scraper)HTML
<a href="/listings/yfe404--web-scraper"><img src="/api/listings/yfe404--web-scraper/badge.svg" alt="findsafeskills safety rating" /></a>Install
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
Scenario 1: Scrape a Website
User: "Scrape https://example.com" Claude will automatically: 1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps 2. QUALITY GATE: All data in HTML? → Skip browser, go to validation 3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM 4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs 5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths 6. Phase 4: Protection testing (only if signals detected or user requested) 7. Phase 5: Generate intelligence report with self-critique 8. Implement recommended approach iteratively 9. Test with small batch, then scale
Scenario 2: Create Apify Actor
User: "Make this an Apify Actor" Claude will: 1. Recommend TypeScript (strongly) 2. Guide through `apify create` command 3. Help choose appropriate template (Cheerio vs Playwright) 4. Port scraping logic to Actor format 5. Configure input schema 6. Test and deploy
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 453 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- ---
- name: web-scraping
- description: This skill activates for web scraping and Actor development. It proactively discovers APIs via traffic interception, recommends optimal strategy (traffic interception/sitemap/API/DOM scraping/hybrid), and implements iteratively. For production, it guides TypeScript Actor creation via Apify CLI.
- license: MIT
- ---
- # Web Scraping with Intelligent Strategy Selection
- ## When This Skill Activates
- Activate automatically when user requests:
- - "Scrape [website]"
- - "Extract data from [site]"
- - "Get product information from [URL]"
- - "Find all links/pages on [site]"
- - "I'm getting blocked" or "Getting 403 errors" (loads `strategies/anti-blocking.md`)
- - "Make this an Apify Actor" (loads `apify/` subdirectory)
- - "Productionize this scraper"
- ## Input Parsing
- Determine reconnaissance depth from user request:
- | User Says | Mode | Phases Run |
- |-----------|------|------------|
- | "quick recon", "just check", "what framework" | Quick | Phase 0 only |
- | "scrape X", "extract data from X" (default) | Standard | Phases 0-3 + 5, Phase 4 only if protection signals detected |
- | "full recon", "deep scan", "production scraping" | Full | All phases (0-5) including protection testing |
- Default is Standard mode. Escalate to Full if protection signals appear during any phase.
- ## Adaptive Reconnaissance Workflow
- This skill uses an adaptive phased workflow with quality gates. Each gate asks **"Do I have enough?"** — continue only when the answer is no.
- **See**: `strategies/framework-signatures.md` for framework detection tables referenced throughout.
- ### Phase 0: QUICK ASSESSMENT (curl, no browser)
- Gather maximum intelligence with minimum cost — a single HTTP request.
- **Step 0a: Fetch raw HTML and headers**
- ```bash
- curl -s -D- -L "https://target.com/page" -o response.html
- ```
- **Step 0b: Check response headers**
- - Match headers against `strategies/framework-signatures.md` → Response Header Signatures table
- - Note `Server`, `X-Powered-By`, `X-Shopify-Stage`, `Set-Cookie` (protection markers)
- - Check HTTP status code (200 = accessible, 403 = protected, 3xx = redirects)
- **Step 0c: Check Known Major Sites table**
- - Match domain against `strategies/framework-signatures.md` → Known Major Sites
- - If matched: use the specified data strategy, skip generic pattern scanning
- **Step 0d: Detect framework from HTML**
- - Search raw HTML for signatures in `strategies/framework-signatures.md` → HTML Signatures table
- - Look for `__NEXT_DATA__`, `__NUXT__`, `ld+json`, `/wp-content/`, `data-reactroot`
- **Step 0e: Search for target data points**
- - For each data point the user wants: search raw HTML for that content
- - Track which data points are found vs missing
- - Check for sitemaps: `curl -s https://[site]/robots.txt | grep -i Sitemap`
- **Step 0f: Note protection signals**
- - 403/503 status, Cloudflare challenge HTML, CAPTCHA elements, `cf-ray` header
- - Record for Phase 4 decision
- **See**: `strategies/cheerio-vs-browser-test.md` for the Cheerio viability assessment
- > **QUALITY GATE A**: All target data points found in raw HTML + no protection signals?
- > → YES: Skip to Phase 3 (Validate Findings). No browser needed.
- > → NO: Continue to Phase 1.
- ### Phase 1: BROWSER RECONNAISSANCE (only if Phase 0 needs it)
- Launch browser only for data points missing from raw HTML or when JavaScript rendering is required.
- **Step 1a: Initialize browser session**
- - `proxy_start()` → Start traffic interception proxy
- - `interceptor_chrome_launch(url, stealthMode: true)` → Launch Chrome with anti-detection
- - `interceptor_chrome_devtools_attach(target_id)` → Attach DevTools bridge
- - `interceptor_chrome_devtools_screenshot()` → Capture visual state
- **Step 1b: Capture traffic and rendered DOM**
- - `proxy_list_traffic()` → Review all traffic from page load
- - `proxy_search_traffic(query: "application/json")` → Find JSON responses
- - `interceptor_chrome_devtools_list_network(resource_types: ["xhr", "fetch"])` → XHR/fetch calls
- - `interceptor_chrome_devtools_snapshot()` → Accessibility tree (rendered DOM)
- **Step 1c: Search rendered DOM for missing data points**
- - For each data point NOT found in Phase 0: search rendered DOM
- - Use framework-specific search strategy from `strategies/framework-signatures.md` → Framework → Search Strategy table
- - Only search patterns relevant to the detected framework
- **Step 1d: Inspect discovered endpoints**
- - `proxy_get_exchange(exchange_id)` → Full request/response for promising endpoints
- - Document: method, headers, auth, response structure, pagination
- > **QUALITY GATE B**: All target data points now covered (raw HTML + rendered DOM + traffic)?
- > → YES: Skip to Phase 3 (Validate Findings). No deep scan needed.
- > → NO: Continue to Phase 2 for missing data points only.
- ### Phase 2: DEEP SCAN (only for missing data points)
- Targeted investigation for data points not yet found. Only search for what's missing.
- **Step 2a: Test interactions for missing data**
- - `proxy_clear_traffic()` before each action → Isolate API calls
- - `humanizer_click(target_id, selector)` → Trigger dynamic content loads
- - `humanizer_scroll(target_id, direction, amount)` → Trigger lazy loading / infinite scroll
- - `humanizer_idle(target_id, duration_ms)` → Wait for delayed content
- - After each action: `proxy_list_traffic()` → Check for new API calls
- **Step 2b: Sniff APIs (framework-aware)**
- - Search only patterns relevant to detected framework:
- - Next.js → `proxy_list_traffic(url_filter: "/_next/data/")`
- - WordPress → `proxy_list_traffic(url_filter: "/wp-json/")`
- - GraphQL → `proxy_search_traffic(query: "graphql")`
- - Generic → `proxy_list_traffic(url_filter: "/api/")` + `proxy_search_traffic(query: "application/json")`
- - Skip patterns that don't apply to the detected framework
- **Step 2c: Test pagination and filtering**
- - Only if pagination data is a missing data point or needed for coverage assessment
- - `proxy_clear_traffic()` → click next page → `proxy_list_traffic(url_filter: "page=")`
- - Document pagination type (URL-based, API offset, cursor, infinite scroll)
- > **QUALITY GATE C**: Enough data points covered for a useful report?
- > → YES: Go to Phase 3.
- > → NO: Document gaps, go to Phase 3 anyway (report will note missing data in self-critique).
- ### Phase 3: VALIDATE FINDINGS
- Every claimed extraction method must be verified. A data point is not "found" until the extraction path is specified and tested.
- **See**: `strategies/cheerio-vs-browser-test.md` for validation methodology
- **Step 3a: Validate CSS selectors**
- - For each Cheerio/selector-based method: confirm the selector matches actual HTML
- - Test against raw HTML (curl output) or rendered DOM (snapshot)
- - Confirm selector extracts the correct value, not a different element
- **Step 3b: Validate JSON paths**
- - For each JSON extraction (e.g., `__NEXT_DATA__`, API response): confirm the path resolves
- - Parse the JSON, follow the path, verify it returns the expected data type and value
- **Step 3c: Validate API endpoints**
- - For each discovered API: replay the request (curl or `proxy_get_exchange`)
- - Confirm: response status 200, expected data structure, correct values
- - Test pagination if claimed (at least page 1 and page 2)
- **Step 3d: Downgrade or re-investigate failures**
- - If a selector doesn't match: try alternative selectors, or downgrade to PARTIAL confidence
- - If an API returns 403: note protection requirement, flag for Phase 4
- - If a JSON path is wrong: re-examine the JSON structure, correct the path
- ### Phase 4: PROTECTION TESTING (conditional)
- **See**: `strategies/proxy-escalation.md` for complete skip/run decision logic
- **Skip Phase 4 when ALL true**:
- - No protection signals detected in Phases 0-2
- - All data points have validated extraction methods
- - User didn't request "full recon"
- **Run Phase 4 when ANY true**:
- - 403/challenge page observed during any phase
- - Known high-protection domain
- - High-volume or production intent
- - User explicitly requested it
- **If running**:
- **Step 4a: Test raw HTTP access**
- ```bash
- curl -s -o /dev/null -w "%{http_code}" "https://target.com/page"
- ```
- - 200 → Cheerio viable, no browser needed for accessible endpoints
- - 403/503 → Escalate to stealth browser
- **Step 4b: Test with stealth browser** (if needed)
- - Already running from Phase 1 — check if pages loaded without challenges
- - `interceptor_chrome_devtools_list_cookies(domain_filter: "cloudflare")` → Protection cookies
- - `interceptor_chrome_devtools_list_storage_keys(storage_type: "local")` → Fingerprint markers
- - `proxy_get_tls_fingerprints()` → TLS fingerprint analysis
- **Step 4c: Test with upstream proxy** (if needed)
- - `proxy_set_upstream("http://user:pass@proxy-provider:port")`
- - Re-test blocked endpoints through proxy
- - Document minimum access level for each data point
- **Step 4d: Document protection profile**
- - What protections exist, what worked to bypass them, what production scrapers will need
- ### Phase 5: REPORT + SELF-CRITIQUE
- Generate the intelligence report, then critically review it for gaps.
- **See**: `reference/report-schema.md` for complete report format
- **Step 5a: Generate report**
- - Follow `reference/report-schema.md` schema (Sections 1-6)
- - Include `Validated?` status for every strategy (YES / PARTIAL / NO)
- - Include all discovered endpoints with full specs
- **Step 5b: Self-critique**
- - Write Section 7 (Self-Critique) per `reference/report-schema.md`:
- - **Gaps**: Data points not found — why, and what would find them
- - **Skipped steps**: Which phases skipped, with quality gate reasoning
- - **Unvalidated claims**: Anything marked PARTIAL or NO
- - **Assumptions**: Things not verified (e.g., "consistent layout across categories")
- - **Staleness risk**: Geo-dependent prices, A/B layouts, session-specific content
- - **Recommendations**: Targeted next steps (not "re-run everything")
- **Step 5c: Fix gaps with targeted re-investigation**
- - If self-critique reveals fixable gaps: go back to the specific phase/step, not a full re-run
- - Example: "Price selector untested" → run one curl + parse, don't re-launch browser
- - Update report with results
- **Step 5d: Record session** (if browser was used)
- - `proxy_session_start(name)` → `proxy_session_stop(session_id)` → `proxy_export_har(session_id, path)`
- - HAR file captures all traffic for replay. See `strategies/session-workflows.md`
- ---
- ### IMPLEMENTATION (after reconnaissance)
- After reconnaissance report is accepted, implement scraper iteratively.
- **Core Pattern**:
- 1. Implement recommended approach (minimal code)
- 2. Test with small batch (5-10 items)
- 3. Validate data quality
- 4. Scale to full dataset or fallback
- 5. Handle blocking if encountered
- 6. Add robustness (error handling, retries, logging)
- **See**: `workflows/implementation.md` for complete implementation patterns and code examples
- ### PRODUCTIONIZATION (on request)
- Convert scraper to production-ready Apify Actor.
- **Activation triggers**: "Make this an Apify Actor", "Productionize this", "Deploy to Apify"
- **Core Pattern**:
- 1. Confirm TypeScript preference (STRONGLY RECOMMENDED)
- 2. Initialize with `apify create` command (CRITICAL)
- 3. Port scraping logic to Actor format
- 4. Test locally and deploy
- **Note**: During development, proxy-mcp provides reconnaissance and traffic analysis. For production Actors, use Crawlee crawlers (CheerioCrawler/PlaywrightCrawler) on Apify infrastructure.
- **See**: `workflows/productionization.md` for complete workflow and `apify/` for Actor development guides
- ## Quick Reference
- | Task | Pattern/Command | Documentation |
- |------|----------------|---------------|
- | **Reconnaissance** | **Adaptive Phases 0-5** | **`workflows/reconnaissance.md`** |
- | Framework detection | Header + HTML signature matching | `strategies/framework-signatures.md` |
- | Cheerio vs Browser | Three-way test + early exit | `strategies/cheerio-vs-browser-test.md` |
- | Traffic analysis | `proxy_list_traffic()` + `proxy_get_exchange()` | `strategies/traffic-interception.md` |
- | Protection testing | Conditional escalation | `strategies/proxy-escalation.md` |
- | Report format | Sections 1-7 with self-critique | `reference/report-schema.md` |
- | Find sitemaps | `RobotsFile.find(url)` | `strategies/sitemap-discovery.md` |
- | Filter sitemap URLs | `RequestList + regex` | `reference/regex-patterns.md` |
- | Discover APIs | Traffic capture (automatic) | `strategies/api-discovery.md` |
- | DOM scraping | DevTools bridge + humanizer | `strategies/dom-scraping.md` |
- | HTTP scraping | `CheerioCrawler` | `strategies/cheerio-scraping.md` |
- | Hybrid approach | Sitemap + API | `strategies/hybrid-approaches.md` |
- | Handle blocking | Stealth mode + upstream proxies | `strategies/anti-blocking.md` |
- | Session recording | `proxy_session_start()` / `proxy_export_har()` | `strategies/session-workflows.md` |
- | Proxy-MCP tools | Complete reference | `reference/proxy-tool-reference.md` |
- | Fingerprint configs | Stealth + TLS presets | `reference/fingerprint-patterns.md` |
- | Create Apify Actor | `apify create` | `apify/cli-workflow.md` |
- | Template selection | Cheerio vs Playwright | `workflows/productionization.md` |
- | Input schema | `.actor/input_schema.json` | `apify/input-schemas.md` |
- | Deploy actor | `apify push` | `apify/deployment.md` |
- ## Common Patterns
- ### Pattern 1: Sitemap-Based Scraping
- ```javascript
- import { RobotsFile, CheerioCrawler, Dataset } from 'crawlee';
- // Auto-discover and parse sitemaps
- const robots = await RobotsFile.find('https://example.com');
- const urls = await robots.parseUrlsFromSitemaps();
- const crawler = new CheerioCrawler({
- async requestHandler({ $, request }) {
- const data = {
- title: $('h1').text().trim(),
- // ... extract data
- };
- await Dataset.pushData(data);
- },
- });
- await crawler.addRequests(urls);
- await crawler.run();
- ```
- See `examples/sitemap-basic.js` for complete example.
- ### Pattern 2: API-Based Scraping
- ```javascript
- import { gotScraping } from 'got-scraping';
- const productIds = [123, 456, 789];
- for (const id of productIds) {
- const response = await gotScraping({
- url: `https://api.example.com/products/${id}`,
- responseType: 'json',
- });
- console.log(response.body);
- }
- ```
- See `examples/api-scraper.js` for complete example.
- ### Pattern 3: Hybrid (Sitemap + API)
- ```javascript
- // Get URLs from sitemap
- const robots = await RobotsFile.find('https://shop.com');
- const urls = await robots.parseUrlsFromSitemaps();
- // Extract IDs from URLs
- const productIds = urls
- .map(url => url.match(/\/products\/(\d+)/)?.[1])
- .filter(Boolean);
- // Fetch data via API
- for (const id of productIds) {
- const data = await gotScraping({
- url: `https://api.shop.com/v1/products/${id}`,
- responseType: 'json',
- });
- // Process data
- }
- ```
- See `examples/hybrid-sitemap-api.js` for complete example.
- ## Directory Navigation
- This skill uses **progressive disclosure** - detailed information is organized in subdirectories and loaded only when needed.
- ### Workflows (Implementation Patterns)
- **For**: Step-by-step workflow guides for each phase
- - `workflows/reconnaissance.md` - **Phase 1 interactive reconnaissance (CRITICAL)**
- - `workflows/implementation.md` - Phase 4 iterative implementation patterns
- - `workflows/productionization.md` - Phase 5 Apify Actor creation workflow
- ### Strategies (Deep Dives)
- **For**: Detailed guides on specific scraping approaches
- - `strategies/framework-signatures.md` - **Framework detection lookup tables (Phase 0/1)**
- - `strategies/cheerio-vs-browser-test.md` - **Cheerio vs Browser decision test with early exit**
- - `strategies/proxy-escalation.md` - **Protection testing skip/run conditions (Phase 4)**
- - `strategies/traffic-interception.md` - Traffic interception via MITM proxy
- - `strategies/sitemap-discovery.md` - Complete sitemap guide (4 patterns)
- - `strategies/api-discovery.md` - Finding and using APIs
- - `strategies/dom-scraping.md` - DOM scraping via DevTools bridge
- - `strategies/cheerio-scraping.md` - HTTP-only scraping
- - `strategies/hybrid-approaches.md` - Combining strategies
- - `strategies/anti-blocking.md` - Multi-layer anti-detection (stealth, humanizer, proxies, TLS)
- - `strategies/session-workflows.md` - Session recording, HAR export, replay
- ### Examples (Runnable Code)
- **For**: Working code to reference or execute
- **JavaScript Learning Examples** (Simple standalone scripts):
- - `examples/sitemap-basic.js` - Simple sitemap scraper
- - `examples/api-scraper.js` - Pure API approach
- - `examples/traffic-interception-basic.js` - Proxy-based reconnaissance
- - `examples/hybrid-sitemap-api.js` - Combined approach
- - `examples/iterative-fallback.js` - Try traffic interception→sitemap→API→DOM scraping
- **TypeScript Production Examples** (Complete Actors):
- - `apify/examples/basic-scraper/` - Sitemap + Playwright
- - `apify/examples/anti-blocking/` - Fingerprinting + proxies
- - `apify/examples/hybrid-api/` - Sitemap + API (optimal)
- ### Reference (Quick Lookup)
- **For**: Quick patterns and troubleshooting
- - `reference/report-schema.md` - **Intelligence report format (Sections 1-7 + self-critique)**
- - `reference/proxy-tool-reference.md` - Proxy-MCP tool reference (all 80+ tools)
- - `reference/regex-patterns.md` - Common URL regex patterns
- - `reference/fingerprint-patterns.md` - Stealth mode + TLS fingerprint presets
- - `reference/anti-patterns.md` - What NOT to do
- ### Apify (Production Deployment)
- **For**: Creating production Apify Actors
- - `apify/README.md` - When and how to use Apify
- - `apify/typescript-first.md` - **Why TypeScript for actors**
- - `apify/cli-workflow.md` - **apify create workflow (CRITICAL)**
- - `apify/initialization.md` - Complete setup guide
- - `apify/input-schemas.md` - Input validation patterns
- - `apify/configuration.md` - actor.json setup
- - `apify/deployment.md` - Testing and deployment
- - `apify/templates/` - TypeScript boilerplate
- **Note**: Each file is self-contained and can be read independently. Claude will navigate to specific files as needed.
- ## Core Principles
- ### 1. Assess Before Committing Resources
- Start cheap (curl), escalate only when needed:
- - Phase 0 (curl) before Phase 1 (browser) before Phase 2 (deep scan)
- - Quality gates skip phases when data is sufficient
- - Never launch a browser if curl gives you everything
- ### 2. Detect First, Then Search Relevant Patterns
- Use framework detection to focus searches:
- - Match against `strategies/framework-signatures.md` before scanning
- - Skip patterns that don't apply (no `__NEXT_DATA__` on Amazon)
- - Known major sites get direct strategy lookup
- ### 3. Validate, Don't Assume
- Every claimed extraction method must be tested:
- - "Found text in HTML" is not enough — need a working selector/path
- - Phase 3 validates every finding before the report
- - Unvalidated claims are marked PARTIAL or NO in the report
- ### 4. Iterative Implementation
- Build incrementally:
- - Small test batch first (5-10 items)
- - Validate quality
- - Scale or fallback
- - Add robustness last
- ### 5. Production-Ready Code
- When productionizing:
- - Use TypeScript (strongly recommended)
- - Use `apify create` (never manual setup)
- - Add proper error handling
- - Include logging and monitoring
- ---
- **Remember**: Traffic interception first, sitemaps second, APIs third, DOM scraping last!
- For detailed guidance on any topic, navigate to the relevant subdirectory file listed above.
README.md · README · 347 lines
View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)
- # Web Scraping Skill
- Intelligent web scraping with automatic strategy selection and TypeScript-first Apify Actor development.
- ## Overview
- This skill provides:
- - **Adaptive reconnaissance** - Phases 0-5 with quality gates that skip unnecessary work (curl first, browser only if needed)
- - **Framework-aware detection** - Identifies site framework before searching, skips irrelevant patterns
- - **Validated findings** - Every claimed selector/path/API is tested before reporting
- - **Self-critiquing reports** - Intelligence reports include gap analysis and staleness warnings
- - **Iterative implementation** - Starts simple, adds complexity only if needed
- - **Production-ready guidance** - TypeScript-first Apify Actor development
- ## Installation
- Add this skill to Claude Code by placing this directory in the skills folder.
- ## Quick Start
- ### Scenario 1: Scrape a Website
- ```
- User: "Scrape https://example.com"
- Claude will automatically:
- 1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps
- 2. QUALITY GATE: All data in HTML? → Skip browser, go to validation
- 3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM
- 4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs
- 5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths
- 6. Phase 4: Protection testing (only if signals detected or user requested)
- 7. Phase 5: Generate intelligence report with self-critique
- 8. Implement recommended approach iteratively
- 9. Test with small batch, then scale
- ```
- ### Scenario 2: Create Apify Actor
- ```
- User: "Make this an Apify Actor"
- Claude will:
- 1. Recommend TypeScript (strongly)
- 2. Guide through `apify create` command
- 3. Help choose appropriate template (Cheerio vs Playwright)
- 4. Port scraping logic to Actor format
- 5. Configure input schema
- 6. Test and deploy
- ```
- ## Directory Structure
- ```
- web-scraping/
- ├── SKILL.md # Main entry point (proactive workflow)
- ├── workflows/ # Implementation patterns
- │ ├── reconnaissance.md # Phase 1 interactive reconnaissance (CRITICAL)
- │ ├── implementation.md # Phase 4 iterative implementation
- │ └── productionization.md # Phase 5 Actor creation
- ├── strategies/ # Deep-dive guides
- │ ├── framework-signatures.md # Framework detection lookup tables
- │ ├── cheerio-vs-browser-test.md # Cheerio vs Browser decision + early exit
- │ ├── proxy-escalation.md # Protection testing skip/run conditions
- │ ├── traffic-interception.md # MITM proxy traffic capture
- │ ├── sitemap-discovery.md # 60x faster URL discovery
- │ ├── api-discovery.md # 10-100x faster than scraping
- │ ├── dom-scraping.md # DevTools bridge + humanizer
- │ ├── cheerio-scraping.md # HTTP-only (5x faster)
- │ ├── hybrid-approaches.md # Combining strategies
- │ ├── anti-blocking.md # Multi-layer anti-detection
- │ └── session-workflows.md # Session recording, HAR, replay
- ├── examples/ # Runnable code
- │ ├── traffic-interception-basic.js
- │ ├── sitemap-basic.js
- │ ├── api-scraper.js
- │ ├── hybrid-sitemap-api.js
- │ └── iterative-fallback.js
- ├── reference/ # Quick lookup
- │ ├── report-schema.md # Intelligence report format (Sections 1-7)
- │ ├── proxy-tool-reference.md # Proxy-MCP tools (80+)
- │ ├── regex-patterns.md
- │ ├── fingerprint-patterns.md
- │ └── anti-patterns.md
- ├── apify/ # Production deployment
- │ ├── typescript-first.md # Why TypeScript
- │ ├── cli-workflow.md # apify create (CRITICAL)
- │ ├── templates/ # TypeScript boilerplate
- │ └── examples/ # Working actors
- └── README.md # This file
- ```
- ## Best Practices Applied
- This skill follows Anthropic's official best practices for skill development:
- ### 1. Progressive Disclosure Architecture ✓
- **Pattern**: Three-level loading system to manage context efficiently
- - **Level 1**: YAML frontmatter (~85 tokens) - Always loaded
- - **Level 2**: Main SKILL.md (~356 lines) - Loaded when skill invoked
- - **Level 3**: Subdirectories - Loaded on-demand as needed
- **Result**: 70-80% token reduction vs monolithic documentation
- **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#progressive-disclosure-design-principle)
- ### 2. Imperative/Infinitive Form Writing Style ✓
- **Pattern**: Write instructions using verb-first commands, not second-person language
- **Examples**:
- - ✅ "Load this workflow when user requests"
- - ✅ "Check for sitemaps automatically"
- - ❌ "You should load this workflow"
- - ❌ "You need to check for sitemaps"
- **Exception**: Second-person is acceptable in user-facing prompts, code comments, and tutorial examples
- **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
- ### 3. Clear YAML Frontmatter ✓
- **Pattern**: Concise, specific name and description that determine when Claude invokes the skill
- **Applied**:
- - `name: web-scraping` - Clear, hyphen-case identifier
- - `description:` - Specific about activation triggers and capabilities (189 chars, optimized from 244)
- **Source**: [agent_skills_spec.md](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md#yaml-frontmatter)
- ### 4. Lean SKILL.md with Reference Files ✓
- **Pattern**: Keep only essential procedural instructions in SKILL.md; move detailed information to subdirectories
- **Applied**:
- - SKILL.md: Core 4-phase workflow (~356 lines)
- - `workflows/`: Detailed implementation patterns
- - `strategies/`: Deep-dive guides
- - `examples/`: Runnable code
- - `reference/`: Quick lookup patterns
- - `apify/`: Production deployment guides
- **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#references-references)
- ### 5. Scripts, References, and Assets Organization ✓
- **Pattern**: Separate executable code, documentation, and output resources
- **Applied**:
- - `examples/` - Executable JavaScript learning examples (like scripts/)
- - `workflows/`, `strategies/`, `reference/`, `apify/` - Documentation loaded as needed (like references/)
- - `apify/templates/`, `apify/examples/` - Boilerplate code and templates (like assets/)
- **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#bundled-resources-optional)
- ### 6. Purpose-Driven Skill Scope ✓
- **Pattern**: Create focused skills for specific purposes rather than one skill that does everything
- **Applied**: This skill focuses specifically on web scraping and Apify Actor development, not general web development
- **Source**: [Anthropic Skills Best Practices](https://www.anthropic.com/news/skills)
- ### 7. Objective, Instructional Language ✓
- **Pattern**: Use clear, technical language focused on "what" and "how" rather than persuasive or promotional tone
- **Applied**: Direct technical guidance throughout ("Check for sitemaps", "Implement iteratively") vs. marketing language
- **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
- ## Key Features
- ### 1. Adaptive Reconnaissance (Phases 0-5)
- Quality-gated workflow that skips unnecessary phases:
- - **Phase 0**: curl-based assessment — detect framework, search for data, check protections
- - **Phase 1**: Browser only if needed — stealth Chrome, traffic capture, rendered DOM
- - **Phase 2**: Deep scan only for missing data — targeted interactions, framework-aware API sniffing
- - **Phase 3**: Validate every finding — test selectors, replay APIs, confirm JSON paths
- - **Phase 4**: Protection testing only if signals warrant — conditional escalation
- - **Phase 5**: Self-critiquing report — gaps, assumptions, staleness warnings
- ### 2. Framework-Aware Detection
- Uses `strategies/framework-signatures.md` lookup tables:
- - Response headers → framework identification
- - HTML signatures → data location mapping
- - Known major sites → direct strategy (e.g., Amazon: custom SSR, no JSON-LD)
- - Detect first, then search only relevant patterns
- ### 3. Validated Intelligence Reports
- Reports follow `reference/report-schema.md` with:
- - `Validated?` column for every extraction strategy (YES / PARTIAL / NO)
- - Self-Critique section: gaps, skipped steps, assumptions, staleness risk
- - Targeted re-investigation for fixable gaps
- ### 4. Iterative Implementation (Phase 4)
- - Start with simplest approach
- - Test small batch (5-10 items)
- - Scale or fallback based on results
- - Add robustness last
- ### 5. TypeScript-First Apify (Phase 5)
- For production actors:
- - **Strongly recommend** TypeScript
- - **Always use** `apify create` command
- - **Choose template** based on site type (Cheerio for static, Playwright for JS-heavy)
- - Type-safe input/output
- ## Example Workflows
- ### Workflow 1: Unknown Site
- ```
- 1. User: "Scrape example.com"
- 2. Phase 0: curl raw HTML → detect Next.js (__NEXT_DATA__), find product data in JSON
- 3. GATE A: All data in __NEXT_DATA__? → YES → Skip browser
- 4. Phase 3: Validate JSON paths resolve to expected values
- 5. Phase 5: Generate report with self-critique
- 6. Result: No browser needed, Cheerio + JSON parsing sufficient
- ```
- ### Workflow 1b: Site Needing Browser
- ```
- 1. User: "Scrape protected-shop.com"
- 2. Phase 0: curl returns 403 → protection detected, no data in HTML
- 3. GATE A: NO → Continue to Phase 1
- 4. Phase 1: Stealth browser loads page, traffic reveals API endpoint
- 5. GATE B: All data covered via API → Skip Phase 2
- 6. Phase 3: Replay API request, validate response structure
- 7. Phase 4: Protection testing (403 was detected) → stealth browser + proxy needed
- 8. Phase 5: Report + self-critique
- 9. Implements with discovered API + upstream proxies
- 10. Tests with 10 items, scales to full dataset
- ```
- ### Workflow 2: Make it an Actor
- ```
- 1. User: "Make this an Apify Actor"
- 2. Claude loads apify/ module
- 3. Recommends TypeScript? (Yes)
- 4. Guides through: apify create
- 5. Analyzes site: Static HTML → Selects Cheerio template
- 6. Ports scraping logic to TypeScript
- 7. Adds input schema
- 8. Tests: apify run
- 9. Deploys: apify push
- 10. Result: Production-ready actor
- ```
- ## Performance Benefits
- | Approach | Time (1000 pages) | vs Crawling |
- |----------|-------------------|-------------|
- | Sitemap + API | 5 minutes | 60x faster |
- | Sitemap + Playwright | 20 minutes | 15x faster |
- | API only | 8 minutes | 40x faster |
- | Playwright crawl | 45 minutes | Baseline |
- ## Best Practices Summary
- ### Reconnaissance (Phases 0-5)
- - Start with curl (Phase 0) before launching browser
- - Detect framework first, then search relevant patterns only
- - Quality gates skip phases when data is sufficient
- - Validate every selector/path/API before reporting
- - Self-critique: check for gaps, assumptions, staleness
- - Protection testing only when signals warrant it
- ### Implementation Phase (Phase 4)
- - Start simple (traffic interception → sitemap → API → DOM scraping)
- - Test small batch first
- - Handle errors gracefully
- - Respect rate limits
- ### Production Phase (Phase 5)
- - Use TypeScript for Apify Actors
- - Always use `apify create` command
- - Choose template based on Phase 1 findings (Cheerio vs Playwright)
- - Test locally with `apify run`
- - Deploy with `apify push`
- ## Troubleshooting
- ### "No URLs found in sitemap"
- → See `strategies/sitemap-discovery.md` troubleshooting section
- ### "API requires authentication"
- → See `strategies/api-discovery.md` authentication section
- ### "DOM scraping too slow"
- → See `strategies/dom-scraping.md` and consider API discovered via traffic capture
- ### "Actor deployment fails"
- → See `apify/cli-workflow.md` common issues section
- ## Resources
- - **Main skill**: Read `SKILL.md` for complete workflow
- - **Workflows**: Implementation patterns in `workflows/`
- - **Strategies**: Browse `strategies/` for detailed guides
- - **Examples**: Run code in `examples/` directory
- - **Reference**: Quick lookups in `reference/`
- - **Apify**: Production deployment in `apify/`
- ## Philosophy
- **Intelligence first, implementation second!**
- This skill prioritizes:
- 1. **Reconnaissance** - Understand before coding (APIs > Sitemaps > Scraping)
- 2. **Speed** - Fastest approach that works (API 10-100x faster than HTML)
- 3. **Reliability** - Structured data > HTML parsing
- 4. **Maintainability** - TypeScript, proper tooling
- 5. **Best practices** - Industry standards
- ## Version
- **5.0.0** - Traffic-interception-first scraping:
- - **NEW**: Proxy-MCP integration (MITM traffic interception + stealth browser + humanizer)
- - **NEW**: Automatic API discovery via traffic capture
- - **NEW**: Multi-layer anti-detection (stealth mode, humanizer, upstream proxies, TLS spoofing)
- - **NEW**: Session recording and HAR export/replay
- - Progressive disclosure architecture
- - Proactive strategy discovery
- - TypeScript-first Apify guidance
- - Comprehensive examples
- - Modular organization
- ## References
- All best practices sourced from official Anthropic documentation:
- - [Anthropic Skills Repository](https://github.com/anthropics/skills)
- - [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md)
- - [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md)
- - [Anthropic Skills Announcement](https://www.anthropic.com/news/skills)
- ---
- **Start here**: Read `SKILL.md` for the complete proactive workflow.
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.