Skip to content
Back to search

web-scraping

SkillWorks with Claude CodeGitHub

This skill activates for web scraping and Actor development. It proactively discovers APIs via traffic interception, recommends optimal strategy (traffic interception/sitemap/API/DOM scraping/hybrid), and implements iteratively. For production, it guides TypeScript Actor creation via Apify CLI.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/yfe404--web-scraper/badge.svg)](/listings/yfe404--web-scraper)

HTML

<a href="/listings/yfe404--web-scraper"><img src="/api/listings/yfe404--web-scraper/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Scenario 1: Scrape a Website

User: "Scrape https://example.com"

Claude will automatically:
1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps
2. QUALITY GATE: All data in HTML? → Skip browser, go to validation
3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM
4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs
5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths
6. Phase 4: Protection testing (only if signals detected or user requested)
7. Phase 5: Generate intelligence report with self-critique
8. Implement recommended approach iteratively
9. Test with small batch, then scale

Scenario 2: Create Apify Actor

User: "Make this an Apify Actor"

Claude will:
1. Recommend TypeScript (strongly)
2. Guide through `apify create` command
3. Help choose appropriate template (Cheerio vs Playwright)
4. Port scraping logic to Actor format
5. Configure input schema
6. Test and deploy

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 453 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. ---
  2. name: web-scraping
  3. description: This skill activates for web scraping and Actor development. It proactively discovers APIs via traffic interception, recommends optimal strategy (traffic interception/sitemap/API/DOM scraping/hybrid), and implements iteratively. For production, it guides TypeScript Actor creation via Apify CLI.
  4. license: MIT
  5. ---
  6. # Web Scraping with Intelligent Strategy Selection
  7. ## When This Skill Activates
  8. Activate automatically when user requests:
  9. - "Scrape [website]"
  10. - "Extract data from [site]"
  11. - "Get product information from [URL]"
  12. - "Find all links/pages on [site]"
  13. - "I'm getting blocked" or "Getting 403 errors" (loads `strategies/anti-blocking.md`)
  14. - "Make this an Apify Actor" (loads `apify/` subdirectory)
  15. - "Productionize this scraper"
  16. ## Input Parsing
  17. Determine reconnaissance depth from user request:
  18. | User Says | Mode | Phases Run |
  19. |-----------|------|------------|
  20. | "quick recon", "just check", "what framework" | Quick | Phase 0 only |
  21. | "scrape X", "extract data from X" (default) | Standard | Phases 0-3 + 5, Phase 4 only if protection signals detected |
  22. | "full recon", "deep scan", "production scraping" | Full | All phases (0-5) including protection testing |
  23. Default is Standard mode. Escalate to Full if protection signals appear during any phase.
  24. ## Adaptive Reconnaissance Workflow
  25. This skill uses an adaptive phased workflow with quality gates. Each gate asks **"Do I have enough?"** — continue only when the answer is no.
  26. **See**: `strategies/framework-signatures.md` for framework detection tables referenced throughout.
  27. ### Phase 0: QUICK ASSESSMENT (curl, no browser)
  28. Gather maximum intelligence with minimum cost — a single HTTP request.
  29. **Step 0a: Fetch raw HTML and headers**
  30. ```bash
  31. curl -s -D- -L "https://target.com/page" -o response.html
  32. ```
  33. **Step 0b: Check response headers**
  34. - Match headers against `strategies/framework-signatures.md` → Response Header Signatures table
  35. - Note `Server`, `X-Powered-By`, `X-Shopify-Stage`, `Set-Cookie` (protection markers)
  36. - Check HTTP status code (200 = accessible, 403 = protected, 3xx = redirects)
  37. **Step 0c: Check Known Major Sites table**
  38. - Match domain against `strategies/framework-signatures.md` → Known Major Sites
  39. - If matched: use the specified data strategy, skip generic pattern scanning
  40. **Step 0d: Detect framework from HTML**
  41. - Search raw HTML for signatures in `strategies/framework-signatures.md` → HTML Signatures table
  42. - Look for `__NEXT_DATA__`, `__NUXT__`, `ld+json`, `/wp-content/`, `data-reactroot`
  43. **Step 0e: Search for target data points**
  44. - For each data point the user wants: search raw HTML for that content
  45. - Track which data points are found vs missing
  46. - Check for sitemaps: `curl -s https://[site]/robots.txt | grep -i Sitemap`
  47. **Step 0f: Note protection signals**
  48. - 403/503 status, Cloudflare challenge HTML, CAPTCHA elements, `cf-ray` header
  49. - Record for Phase 4 decision
  50. **See**: `strategies/cheerio-vs-browser-test.md` for the Cheerio viability assessment
  51. > **QUALITY GATE A**: All target data points found in raw HTML + no protection signals?
  52. > → YES: Skip to Phase 3 (Validate Findings). No browser needed.
  53. > → NO: Continue to Phase 1.
  54. ### Phase 1: BROWSER RECONNAISSANCE (only if Phase 0 needs it)
  55. Launch browser only for data points missing from raw HTML or when JavaScript rendering is required.
  56. **Step 1a: Initialize browser session**
  57. - `proxy_start()` → Start traffic interception proxy
  58. - `interceptor_chrome_launch(url, stealthMode: true)` → Launch Chrome with anti-detection
  59. - `interceptor_chrome_devtools_attach(target_id)` → Attach DevTools bridge
  60. - `interceptor_chrome_devtools_screenshot()` → Capture visual state
  61. **Step 1b: Capture traffic and rendered DOM**
  62. - `proxy_list_traffic()` → Review all traffic from page load
  63. - `proxy_search_traffic(query: "application/json")` → Find JSON responses
  64. - `interceptor_chrome_devtools_list_network(resource_types: ["xhr", "fetch"])` → XHR/fetch calls
  65. - `interceptor_chrome_devtools_snapshot()` → Accessibility tree (rendered DOM)
  66. **Step 1c: Search rendered DOM for missing data points**
  67. - For each data point NOT found in Phase 0: search rendered DOM
  68. - Use framework-specific search strategy from `strategies/framework-signatures.md` → Framework → Search Strategy table
  69. - Only search patterns relevant to the detected framework
  70. **Step 1d: Inspect discovered endpoints**
  71. - `proxy_get_exchange(exchange_id)` → Full request/response for promising endpoints
  72. - Document: method, headers, auth, response structure, pagination
  73. > **QUALITY GATE B**: All target data points now covered (raw HTML + rendered DOM + traffic)?
  74. > → YES: Skip to Phase 3 (Validate Findings). No deep scan needed.
  75. > → NO: Continue to Phase 2 for missing data points only.
  76. ### Phase 2: DEEP SCAN (only for missing data points)
  77. Targeted investigation for data points not yet found. Only search for what's missing.
  78. **Step 2a: Test interactions for missing data**
  79. - `proxy_clear_traffic()` before each action → Isolate API calls
  80. - `humanizer_click(target_id, selector)` → Trigger dynamic content loads
  81. - `humanizer_scroll(target_id, direction, amount)` → Trigger lazy loading / infinite scroll
  82. - `humanizer_idle(target_id, duration_ms)` → Wait for delayed content
  83. - After each action: `proxy_list_traffic()` → Check for new API calls
  84. **Step 2b: Sniff APIs (framework-aware)**
  85. - Search only patterns relevant to detected framework:
  86. - Next.js → `proxy_list_traffic(url_filter: "/_next/data/")`
  87. - WordPress → `proxy_list_traffic(url_filter: "/wp-json/")`
  88. - GraphQL → `proxy_search_traffic(query: "graphql")`
  89. - Generic → `proxy_list_traffic(url_filter: "/api/")` + `proxy_search_traffic(query: "application/json")`
  90. - Skip patterns that don't apply to the detected framework
  91. **Step 2c: Test pagination and filtering**
  92. - Only if pagination data is a missing data point or needed for coverage assessment
  93. - `proxy_clear_traffic()` → click next page → `proxy_list_traffic(url_filter: "page=")`
  94. - Document pagination type (URL-based, API offset, cursor, infinite scroll)
  95. > **QUALITY GATE C**: Enough data points covered for a useful report?
  96. > → YES: Go to Phase 3.
  97. > → NO: Document gaps, go to Phase 3 anyway (report will note missing data in self-critique).
  98. ### Phase 3: VALIDATE FINDINGS
  99. Every claimed extraction method must be verified. A data point is not "found" until the extraction path is specified and tested.
  100. **See**: `strategies/cheerio-vs-browser-test.md` for validation methodology
  101. **Step 3a: Validate CSS selectors**
  102. - For each Cheerio/selector-based method: confirm the selector matches actual HTML
  103. - Test against raw HTML (curl output) or rendered DOM (snapshot)
  104. - Confirm selector extracts the correct value, not a different element
  105. **Step 3b: Validate JSON paths**
  106. - For each JSON extraction (e.g., `__NEXT_DATA__`, API response): confirm the path resolves
  107. - Parse the JSON, follow the path, verify it returns the expected data type and value
  108. **Step 3c: Validate API endpoints**
  109. - For each discovered API: replay the request (curl or `proxy_get_exchange`)
  110. - Confirm: response status 200, expected data structure, correct values
  111. - Test pagination if claimed (at least page 1 and page 2)
  112. **Step 3d: Downgrade or re-investigate failures**
  113. - If a selector doesn't match: try alternative selectors, or downgrade to PARTIAL confidence
  114. - If an API returns 403: note protection requirement, flag for Phase 4
  115. - If a JSON path is wrong: re-examine the JSON structure, correct the path
  116. ### Phase 4: PROTECTION TESTING (conditional)
  117. **See**: `strategies/proxy-escalation.md` for complete skip/run decision logic
  118. **Skip Phase 4 when ALL true**:
  119. - No protection signals detected in Phases 0-2
  120. - All data points have validated extraction methods
  121. - User didn't request "full recon"
  122. **Run Phase 4 when ANY true**:
  123. - 403/challenge page observed during any phase
  124. - Known high-protection domain
  125. - High-volume or production intent
  126. - User explicitly requested it
  127. **If running**:
  128. **Step 4a: Test raw HTTP access**
  129. ```bash
  130. curl -s -o /dev/null -w "%{http_code}" "https://target.com/page"
  131. ```
  132. - 200 → Cheerio viable, no browser needed for accessible endpoints
  133. - 403/503 → Escalate to stealth browser
  134. **Step 4b: Test with stealth browser** (if needed)
  135. - Already running from Phase 1 — check if pages loaded without challenges
  136. - `interceptor_chrome_devtools_list_cookies(domain_filter: "cloudflare")` → Protection cookies
  137. - `interceptor_chrome_devtools_list_storage_keys(storage_type: "local")` → Fingerprint markers
  138. - `proxy_get_tls_fingerprints()` → TLS fingerprint analysis
  139. **Step 4c: Test with upstream proxy** (if needed)
  140. - `proxy_set_upstream("http://user:pass@proxy-provider:port")`
  141. - Re-test blocked endpoints through proxy
  142. - Document minimum access level for each data point
  143. **Step 4d: Document protection profile**
  144. - What protections exist, what worked to bypass them, what production scrapers will need
  145. ### Phase 5: REPORT + SELF-CRITIQUE
  146. Generate the intelligence report, then critically review it for gaps.
  147. **See**: `reference/report-schema.md` for complete report format
  148. **Step 5a: Generate report**
  149. - Follow `reference/report-schema.md` schema (Sections 1-6)
  150. - Include `Validated?` status for every strategy (YES / PARTIAL / NO)
  151. - Include all discovered endpoints with full specs
  152. **Step 5b: Self-critique**
  153. - Write Section 7 (Self-Critique) per `reference/report-schema.md`:
  154. - **Gaps**: Data points not found — why, and what would find them
  155. - **Skipped steps**: Which phases skipped, with quality gate reasoning
  156. - **Unvalidated claims**: Anything marked PARTIAL or NO
  157. - **Assumptions**: Things not verified (e.g., "consistent layout across categories")
  158. - **Staleness risk**: Geo-dependent prices, A/B layouts, session-specific content
  159. - **Recommendations**: Targeted next steps (not "re-run everything")
  160. **Step 5c: Fix gaps with targeted re-investigation**
  161. - If self-critique reveals fixable gaps: go back to the specific phase/step, not a full re-run
  162. - Example: "Price selector untested" → run one curl + parse, don't re-launch browser
  163. - Update report with results
  164. **Step 5d: Record session** (if browser was used)
  165. - `proxy_session_start(name)` → `proxy_session_stop(session_id)` → `proxy_export_har(session_id, path)`
  166. - HAR file captures all traffic for replay. See `strategies/session-workflows.md`
  167. ---
  168. ### IMPLEMENTATION (after reconnaissance)
  169. After reconnaissance report is accepted, implement scraper iteratively.
  170. **Core Pattern**:
  171. 1. Implement recommended approach (minimal code)
  172. 2. Test with small batch (5-10 items)
  173. 3. Validate data quality
  174. 4. Scale to full dataset or fallback
  175. 5. Handle blocking if encountered
  176. 6. Add robustness (error handling, retries, logging)
  177. **See**: `workflows/implementation.md` for complete implementation patterns and code examples
  178. ### PRODUCTIONIZATION (on request)
  179. Convert scraper to production-ready Apify Actor.
  180. **Activation triggers**: "Make this an Apify Actor", "Productionize this", "Deploy to Apify"
  181. **Core Pattern**:
  182. 1. Confirm TypeScript preference (STRONGLY RECOMMENDED)
  183. 2. Initialize with `apify create` command (CRITICAL)
  184. 3. Port scraping logic to Actor format
  185. 4. Test locally and deploy
  186. **Note**: During development, proxy-mcp provides reconnaissance and traffic analysis. For production Actors, use Crawlee crawlers (CheerioCrawler/PlaywrightCrawler) on Apify infrastructure.
  187. **See**: `workflows/productionization.md` for complete workflow and `apify/` for Actor development guides
  188. ## Quick Reference
  189. | Task | Pattern/Command | Documentation |
  190. |------|----------------|---------------|
  191. | **Reconnaissance** | **Adaptive Phases 0-5** | **`workflows/reconnaissance.md`** |
  192. | Framework detection | Header + HTML signature matching | `strategies/framework-signatures.md` |
  193. | Cheerio vs Browser | Three-way test + early exit | `strategies/cheerio-vs-browser-test.md` |
  194. | Traffic analysis | `proxy_list_traffic()` + `proxy_get_exchange()` | `strategies/traffic-interception.md` |
  195. | Protection testing | Conditional escalation | `strategies/proxy-escalation.md` |
  196. | Report format | Sections 1-7 with self-critique | `reference/report-schema.md` |
  197. | Find sitemaps | `RobotsFile.find(url)` | `strategies/sitemap-discovery.md` |
  198. | Filter sitemap URLs | `RequestList + regex` | `reference/regex-patterns.md` |
  199. | Discover APIs | Traffic capture (automatic) | `strategies/api-discovery.md` |
  200. | DOM scraping | DevTools bridge + humanizer | `strategies/dom-scraping.md` |
  201. | HTTP scraping | `CheerioCrawler` | `strategies/cheerio-scraping.md` |
  202. | Hybrid approach | Sitemap + API | `strategies/hybrid-approaches.md` |
  203. | Handle blocking | Stealth mode + upstream proxies | `strategies/anti-blocking.md` |
  204. | Session recording | `proxy_session_start()` / `proxy_export_har()` | `strategies/session-workflows.md` |
  205. | Proxy-MCP tools | Complete reference | `reference/proxy-tool-reference.md` |
  206. | Fingerprint configs | Stealth + TLS presets | `reference/fingerprint-patterns.md` |
  207. | Create Apify Actor | `apify create` | `apify/cli-workflow.md` |
  208. | Template selection | Cheerio vs Playwright | `workflows/productionization.md` |
  209. | Input schema | `.actor/input_schema.json` | `apify/input-schemas.md` |
  210. | Deploy actor | `apify push` | `apify/deployment.md` |
  211. ## Common Patterns
  212. ### Pattern 1: Sitemap-Based Scraping
  213. ```javascript
  214. import { RobotsFile, CheerioCrawler, Dataset } from 'crawlee';
  215. // Auto-discover and parse sitemaps
  216. const robots = await RobotsFile.find('https://example.com');
  217. const urls = await robots.parseUrlsFromSitemaps();
  218. const crawler = new CheerioCrawler({
  219. async requestHandler({ $, request }) {
  220. const data = {
  221. title: $('h1').text().trim(),
  222. // ... extract data
  223. };
  224. await Dataset.pushData(data);
  225. },
  226. });
  227. await crawler.addRequests(urls);
  228. await crawler.run();
  229. ```
  230. See `examples/sitemap-basic.js` for complete example.
  231. ### Pattern 2: API-Based Scraping
  232. ```javascript
  233. import { gotScraping } from 'got-scraping';
  234. const productIds = [123, 456, 789];
  235. for (const id of productIds) {
  236. const response = await gotScraping({
  237. url: `https://api.example.com/products/${id}`,
  238. responseType: 'json',
  239. });
  240. console.log(response.body);
  241. }
  242. ```
  243. See `examples/api-scraper.js` for complete example.
  244. ### Pattern 3: Hybrid (Sitemap + API)
  245. ```javascript
  246. // Get URLs from sitemap
  247. const robots = await RobotsFile.find('https://shop.com');
  248. const urls = await robots.parseUrlsFromSitemaps();
  249. // Extract IDs from URLs
  250. const productIds = urls
  251. .map(url => url.match(/\/products\/(\d+)/)?.[1])
  252. .filter(Boolean);
  253. // Fetch data via API
  254. for (const id of productIds) {
  255. const data = await gotScraping({
  256. url: `https://api.shop.com/v1/products/${id}`,
  257. responseType: 'json',
  258. });
  259. // Process data
  260. }
  261. ```
  262. See `examples/hybrid-sitemap-api.js` for complete example.
  263. ## Directory Navigation
  264. This skill uses **progressive disclosure** - detailed information is organized in subdirectories and loaded only when needed.
  265. ### Workflows (Implementation Patterns)
  266. **For**: Step-by-step workflow guides for each phase
  267. - `workflows/reconnaissance.md` - **Phase 1 interactive reconnaissance (CRITICAL)**
  268. - `workflows/implementation.md` - Phase 4 iterative implementation patterns
  269. - `workflows/productionization.md` - Phase 5 Apify Actor creation workflow
  270. ### Strategies (Deep Dives)
  271. **For**: Detailed guides on specific scraping approaches
  272. - `strategies/framework-signatures.md` - **Framework detection lookup tables (Phase 0/1)**
  273. - `strategies/cheerio-vs-browser-test.md` - **Cheerio vs Browser decision test with early exit**
  274. - `strategies/proxy-escalation.md` - **Protection testing skip/run conditions (Phase 4)**
  275. - `strategies/traffic-interception.md` - Traffic interception via MITM proxy
  276. - `strategies/sitemap-discovery.md` - Complete sitemap guide (4 patterns)
  277. - `strategies/api-discovery.md` - Finding and using APIs
  278. - `strategies/dom-scraping.md` - DOM scraping via DevTools bridge
  279. - `strategies/cheerio-scraping.md` - HTTP-only scraping
  280. - `strategies/hybrid-approaches.md` - Combining strategies
  281. - `strategies/anti-blocking.md` - Multi-layer anti-detection (stealth, humanizer, proxies, TLS)
  282. - `strategies/session-workflows.md` - Session recording, HAR export, replay
  283. ### Examples (Runnable Code)
  284. **For**: Working code to reference or execute
  285. **JavaScript Learning Examples** (Simple standalone scripts):
  286. - `examples/sitemap-basic.js` - Simple sitemap scraper
  287. - `examples/api-scraper.js` - Pure API approach
  288. - `examples/traffic-interception-basic.js` - Proxy-based reconnaissance
  289. - `examples/hybrid-sitemap-api.js` - Combined approach
  290. - `examples/iterative-fallback.js` - Try traffic interception→sitemap→API→DOM scraping
  291. **TypeScript Production Examples** (Complete Actors):
  292. - `apify/examples/basic-scraper/` - Sitemap + Playwright
  293. - `apify/examples/anti-blocking/` - Fingerprinting + proxies
  294. - `apify/examples/hybrid-api/` - Sitemap + API (optimal)
  295. ### Reference (Quick Lookup)
  296. **For**: Quick patterns and troubleshooting
  297. - `reference/report-schema.md` - **Intelligence report format (Sections 1-7 + self-critique)**
  298. - `reference/proxy-tool-reference.md` - Proxy-MCP tool reference (all 80+ tools)
  299. - `reference/regex-patterns.md` - Common URL regex patterns
  300. - `reference/fingerprint-patterns.md` - Stealth mode + TLS fingerprint presets
  301. - `reference/anti-patterns.md` - What NOT to do
  302. ### Apify (Production Deployment)
  303. **For**: Creating production Apify Actors
  304. - `apify/README.md` - When and how to use Apify
  305. - `apify/typescript-first.md` - **Why TypeScript for actors**
  306. - `apify/cli-workflow.md` - **apify create workflow (CRITICAL)**
  307. - `apify/initialization.md` - Complete setup guide
  308. - `apify/input-schemas.md` - Input validation patterns
  309. - `apify/configuration.md` - actor.json setup
  310. - `apify/deployment.md` - Testing and deployment
  311. - `apify/templates/` - TypeScript boilerplate
  312. **Note**: Each file is self-contained and can be read independently. Claude will navigate to specific files as needed.
  313. ## Core Principles
  314. ### 1. Assess Before Committing Resources
  315. Start cheap (curl), escalate only when needed:
  316. - Phase 0 (curl) before Phase 1 (browser) before Phase 2 (deep scan)
  317. - Quality gates skip phases when data is sufficient
  318. - Never launch a browser if curl gives you everything
  319. ### 2. Detect First, Then Search Relevant Patterns
  320. Use framework detection to focus searches:
  321. - Match against `strategies/framework-signatures.md` before scanning
  322. - Skip patterns that don't apply (no `__NEXT_DATA__` on Amazon)
  323. - Known major sites get direct strategy lookup
  324. ### 3. Validate, Don't Assume
  325. Every claimed extraction method must be tested:
  326. - "Found text in HTML" is not enough — need a working selector/path
  327. - Phase 3 validates every finding before the report
  328. - Unvalidated claims are marked PARTIAL or NO in the report
  329. ### 4. Iterative Implementation
  330. Build incrementally:
  331. - Small test batch first (5-10 items)
  332. - Validate quality
  333. - Scale or fallback
  334. - Add robustness last
  335. ### 5. Production-Ready Code
  336. When productionizing:
  337. - Use TypeScript (strongly recommended)
  338. - Use `apify create` (never manual setup)
  339. - Add proper error handling
  340. - Include logging and monitoring
  341. ---
  342. **Remember**: Traffic interception first, sitemaps second, APIs third, DOM scraping last!
  343. For detailed guidance on any topic, navigate to the relevant subdirectory file listed above.
README.md · README · 347 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Web Scraping Skill
  2. Intelligent web scraping with automatic strategy selection and TypeScript-first Apify Actor development.
  3. ## Overview
  4. This skill provides:
  5. - **Adaptive reconnaissance** - Phases 0-5 with quality gates that skip unnecessary work (curl first, browser only if needed)
  6. - **Framework-aware detection** - Identifies site framework before searching, skips irrelevant patterns
  7. - **Validated findings** - Every claimed selector/path/API is tested before reporting
  8. - **Self-critiquing reports** - Intelligence reports include gap analysis and staleness warnings
  9. - **Iterative implementation** - Starts simple, adds complexity only if needed
  10. - **Production-ready guidance** - TypeScript-first Apify Actor development
  11. ## Installation
  12. Add this skill to Claude Code by placing this directory in the skills folder.
  13. ## Quick Start
  14. ### Scenario 1: Scrape a Website
  15. ```
  16. User: "Scrape https://example.com"
  17. Claude will automatically:
  18. 1. Phase 0: curl raw HTML — detect framework, search for data points, check sitemaps
  19. 2. QUALITY GATE: All data in HTML? → Skip browser, go to validation
  20. 3. Phase 1: Launch stealth browser (only if needed) — capture traffic, rendered DOM
  21. 4. Phase 2: Deep scan (only for missing data) — test interactions, sniff APIs
  22. 5. Phase 3: Validate every finding — test selectors, replay APIs, confirm paths
  23. 6. Phase 4: Protection testing (only if signals detected or user requested)
  24. 7. Phase 5: Generate intelligence report with self-critique
  25. 8. Implement recommended approach iteratively
  26. 9. Test with small batch, then scale
  27. ```
  28. ### Scenario 2: Create Apify Actor
  29. ```
  30. User: "Make this an Apify Actor"
  31. Claude will:
  32. 1. Recommend TypeScript (strongly)
  33. 2. Guide through `apify create` command
  34. 3. Help choose appropriate template (Cheerio vs Playwright)
  35. 4. Port scraping logic to Actor format
  36. 5. Configure input schema
  37. 6. Test and deploy
  38. ```
  39. ## Directory Structure
  40. ```
  41. web-scraping/
  42. ├── SKILL.md # Main entry point (proactive workflow)
  43. ├── workflows/ # Implementation patterns
  44. │ ├── reconnaissance.md # Phase 1 interactive reconnaissance (CRITICAL)
  45. │ ├── implementation.md # Phase 4 iterative implementation
  46. │ └── productionization.md # Phase 5 Actor creation
  47. ├── strategies/ # Deep-dive guides
  48. │ ├── framework-signatures.md # Framework detection lookup tables
  49. │ ├── cheerio-vs-browser-test.md # Cheerio vs Browser decision + early exit
  50. │ ├── proxy-escalation.md # Protection testing skip/run conditions
  51. │ ├── traffic-interception.md # MITM proxy traffic capture
  52. │ ├── sitemap-discovery.md # 60x faster URL discovery
  53. │ ├── api-discovery.md # 10-100x faster than scraping
  54. │ ├── dom-scraping.md # DevTools bridge + humanizer
  55. │ ├── cheerio-scraping.md # HTTP-only (5x faster)
  56. │ ├── hybrid-approaches.md # Combining strategies
  57. │ ├── anti-blocking.md # Multi-layer anti-detection
  58. │ └── session-workflows.md # Session recording, HAR, replay
  59. ├── examples/ # Runnable code
  60. │ ├── traffic-interception-basic.js
  61. │ ├── sitemap-basic.js
  62. │ ├── api-scraper.js
  63. │ ├── hybrid-sitemap-api.js
  64. │ └── iterative-fallback.js
  65. ├── reference/ # Quick lookup
  66. │ ├── report-schema.md # Intelligence report format (Sections 1-7)
  67. │ ├── proxy-tool-reference.md # Proxy-MCP tools (80+)
  68. │ ├── regex-patterns.md
  69. │ ├── fingerprint-patterns.md
  70. │ └── anti-patterns.md
  71. ├── apify/ # Production deployment
  72. │ ├── typescript-first.md # Why TypeScript
  73. │ ├── cli-workflow.md # apify create (CRITICAL)
  74. │ ├── templates/ # TypeScript boilerplate
  75. │ └── examples/ # Working actors
  76. └── README.md # This file
  77. ```
  78. ## Best Practices Applied
  79. This skill follows Anthropic's official best practices for skill development:
  80. ### 1. Progressive Disclosure Architecture ✓
  81. **Pattern**: Three-level loading system to manage context efficiently
  82. - **Level 1**: YAML frontmatter (~85 tokens) - Always loaded
  83. - **Level 2**: Main SKILL.md (~356 lines) - Loaded when skill invoked
  84. - **Level 3**: Subdirectories - Loaded on-demand as needed
  85. **Result**: 70-80% token reduction vs monolithic documentation
  86. **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#progressive-disclosure-design-principle)
  87. ### 2. Imperative/Infinitive Form Writing Style ✓
  88. **Pattern**: Write instructions using verb-first commands, not second-person language
  89. **Examples**:
  90. - ✅ "Load this workflow when user requests"
  91. - ✅ "Check for sitemaps automatically"
  92. - ❌ "You should load this workflow"
  93. - ❌ "You need to check for sitemaps"
  94. **Exception**: Second-person is acceptable in user-facing prompts, code comments, and tutorial examples
  95. **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
  96. ### 3. Clear YAML Frontmatter ✓
  97. **Pattern**: Concise, specific name and description that determine when Claude invokes the skill
  98. **Applied**:
  99. - `name: web-scraping` - Clear, hyphen-case identifier
  100. - `description:` - Specific about activation triggers and capabilities (189 chars, optimized from 244)
  101. **Source**: [agent_skills_spec.md](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md#yaml-frontmatter)
  102. ### 4. Lean SKILL.md with Reference Files ✓
  103. **Pattern**: Keep only essential procedural instructions in SKILL.md; move detailed information to subdirectories
  104. **Applied**:
  105. - SKILL.md: Core 4-phase workflow (~356 lines)
  106. - `workflows/`: Detailed implementation patterns
  107. - `strategies/`: Deep-dive guides
  108. - `examples/`: Runnable code
  109. - `reference/`: Quick lookup patterns
  110. - `apify/`: Production deployment guides
  111. **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#references-references)
  112. ### 5. Scripts, References, and Assets Organization ✓
  113. **Pattern**: Separate executable code, documentation, and output resources
  114. **Applied**:
  115. - `examples/` - Executable JavaScript learning examples (like scripts/)
  116. - `workflows/`, `strategies/`, `reference/`, `apify/` - Documentation loaded as needed (like references/)
  117. - `apify/templates/`, `apify/examples/` - Boilerplate code and templates (like assets/)
  118. **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#bundled-resources-optional)
  119. ### 6. Purpose-Driven Skill Scope ✓
  120. **Pattern**: Create focused skills for specific purposes rather than one skill that does everything
  121. **Applied**: This skill focuses specifically on web scraping and Apify Actor development, not general web development
  122. **Source**: [Anthropic Skills Best Practices](https://www.anthropic.com/news/skills)
  123. ### 7. Objective, Instructional Language ✓
  124. **Pattern**: Use clear, technical language focused on "what" and "how" rather than persuasive or promotional tone
  125. **Applied**: Direct technical guidance throughout ("Check for sitemaps", "Implement iteratively") vs. marketing language
  126. **Source**: [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md#update-skillmd)
  127. ## Key Features
  128. ### 1. Adaptive Reconnaissance (Phases 0-5)
  129. Quality-gated workflow that skips unnecessary phases:
  130. - **Phase 0**: curl-based assessment — detect framework, search for data, check protections
  131. - **Phase 1**: Browser only if needed — stealth Chrome, traffic capture, rendered DOM
  132. - **Phase 2**: Deep scan only for missing data — targeted interactions, framework-aware API sniffing
  133. - **Phase 3**: Validate every finding — test selectors, replay APIs, confirm JSON paths
  134. - **Phase 4**: Protection testing only if signals warrant — conditional escalation
  135. - **Phase 5**: Self-critiquing report — gaps, assumptions, staleness warnings
  136. ### 2. Framework-Aware Detection
  137. Uses `strategies/framework-signatures.md` lookup tables:
  138. - Response headers → framework identification
  139. - HTML signatures → data location mapping
  140. - Known major sites → direct strategy (e.g., Amazon: custom SSR, no JSON-LD)
  141. - Detect first, then search only relevant patterns
  142. ### 3. Validated Intelligence Reports
  143. Reports follow `reference/report-schema.md` with:
  144. - `Validated?` column for every extraction strategy (YES / PARTIAL / NO)
  145. - Self-Critique section: gaps, skipped steps, assumptions, staleness risk
  146. - Targeted re-investigation for fixable gaps
  147. ### 4. Iterative Implementation (Phase 4)
  148. - Start with simplest approach
  149. - Test small batch (5-10 items)
  150. - Scale or fallback based on results
  151. - Add robustness last
  152. ### 5. TypeScript-First Apify (Phase 5)
  153. For production actors:
  154. - **Strongly recommend** TypeScript
  155. - **Always use** `apify create` command
  156. - **Choose template** based on site type (Cheerio for static, Playwright for JS-heavy)
  157. - Type-safe input/output
  158. ## Example Workflows
  159. ### Workflow 1: Unknown Site
  160. ```
  161. 1. User: "Scrape example.com"
  162. 2. Phase 0: curl raw HTML → detect Next.js (__NEXT_DATA__), find product data in JSON
  163. 3. GATE A: All data in __NEXT_DATA__? → YES → Skip browser
  164. 4. Phase 3: Validate JSON paths resolve to expected values
  165. 5. Phase 5: Generate report with self-critique
  166. 6. Result: No browser needed, Cheerio + JSON parsing sufficient
  167. ```
  168. ### Workflow 1b: Site Needing Browser
  169. ```
  170. 1. User: "Scrape protected-shop.com"
  171. 2. Phase 0: curl returns 403 → protection detected, no data in HTML
  172. 3. GATE A: NO → Continue to Phase 1
  173. 4. Phase 1: Stealth browser loads page, traffic reveals API endpoint
  174. 5. GATE B: All data covered via API → Skip Phase 2
  175. 6. Phase 3: Replay API request, validate response structure
  176. 7. Phase 4: Protection testing (403 was detected) → stealth browser + proxy needed
  177. 8. Phase 5: Report + self-critique
  178. 9. Implements with discovered API + upstream proxies
  179. 10. Tests with 10 items, scales to full dataset
  180. ```
  181. ### Workflow 2: Make it an Actor
  182. ```
  183. 1. User: "Make this an Apify Actor"
  184. 2. Claude loads apify/ module
  185. 3. Recommends TypeScript? (Yes)
  186. 4. Guides through: apify create
  187. 5. Analyzes site: Static HTML → Selects Cheerio template
  188. 6. Ports scraping logic to TypeScript
  189. 7. Adds input schema
  190. 8. Tests: apify run
  191. 9. Deploys: apify push
  192. 10. Result: Production-ready actor
  193. ```
  194. ## Performance Benefits
  195. | Approach | Time (1000 pages) | vs Crawling |
  196. |----------|-------------------|-------------|
  197. | Sitemap + API | 5 minutes | 60x faster |
  198. | Sitemap + Playwright | 20 minutes | 15x faster |
  199. | API only | 8 minutes | 40x faster |
  200. | Playwright crawl | 45 minutes | Baseline |
  201. ## Best Practices Summary
  202. ### Reconnaissance (Phases 0-5)
  203. - Start with curl (Phase 0) before launching browser
  204. - Detect framework first, then search relevant patterns only
  205. - Quality gates skip phases when data is sufficient
  206. - Validate every selector/path/API before reporting
  207. - Self-critique: check for gaps, assumptions, staleness
  208. - Protection testing only when signals warrant it
  209. ### Implementation Phase (Phase 4)
  210. - Start simple (traffic interception → sitemap → API → DOM scraping)
  211. - Test small batch first
  212. - Handle errors gracefully
  213. - Respect rate limits
  214. ### Production Phase (Phase 5)
  215. - Use TypeScript for Apify Actors
  216. - Always use `apify create` command
  217. - Choose template based on Phase 1 findings (Cheerio vs Playwright)
  218. - Test locally with `apify run`
  219. - Deploy with `apify push`
  220. ## Troubleshooting
  221. ### "No URLs found in sitemap"
  222. → See `strategies/sitemap-discovery.md` troubleshooting section
  223. ### "API requires authentication"
  224. → See `strategies/api-discovery.md` authentication section
  225. ### "DOM scraping too slow"
  226. → See `strategies/dom-scraping.md` and consider API discovered via traffic capture
  227. ### "Actor deployment fails"
  228. → See `apify/cli-workflow.md` common issues section
  229. ## Resources
  230. - **Main skill**: Read `SKILL.md` for complete workflow
  231. - **Workflows**: Implementation patterns in `workflows/`
  232. - **Strategies**: Browse `strategies/` for detailed guides
  233. - **Examples**: Run code in `examples/` directory
  234. - **Reference**: Quick lookups in `reference/`
  235. - **Apify**: Production deployment in `apify/`
  236. ## Philosophy
  237. **Intelligence first, implementation second!**
  238. This skill prioritizes:
  239. 1. **Reconnaissance** - Understand before coding (APIs > Sitemaps > Scraping)
  240. 2. **Speed** - Fastest approach that works (API 10-100x faster than HTML)
  241. 3. **Reliability** - Structured data > HTML parsing
  242. 4. **Maintainability** - TypeScript, proper tooling
  243. 5. **Best practices** - Industry standards
  244. ## Version
  245. **5.0.0** - Traffic-interception-first scraping:
  246. - **NEW**: Proxy-MCP integration (MITM traffic interception + stealth browser + humanizer)
  247. - **NEW**: Automatic API discovery via traffic capture
  248. - **NEW**: Multi-layer anti-detection (stealth mode, humanizer, upstream proxies, TLS spoofing)
  249. - **NEW**: Session recording and HAR export/replay
  250. - Progressive disclosure architecture
  251. - Proactive strategy discovery
  252. - TypeScript-first Apify guidance
  253. - Comprehensive examples
  254. - Modular organization
  255. ## References
  256. All best practices sourced from official Anthropic documentation:
  257. - [Anthropic Skills Repository](https://github.com/anthropics/skills)
  258. - [Agent Skills Specification](https://github.com/anthropics/skills/blob/main/agent_skills_spec.md)
  259. - [skill-creator/SKILL.md](https://github.com/anthropics/skills/blob/main/skill-creator/SKILL.md)
  260. - [Anthropic Skills Announcement](https://www.anthropic.com/news/skills)
  261. ---
  262. **Start here**: Read `SKILL.md` for the complete proactive workflow.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.