Skip to content
Back to search

x-tweet-fetcher

SkillWorks with Claude CodeGitHub

Fetch tweets, replies, timelines, search results, X Lists, and X Articles from X/Twitter without login or API keys. Single tweets: zero dependencies (FxTwitter). Timelines/search/replies: a Nitter instance (XTF_NITTER). Lists: Nitter first, browser fallback. Articles: a browser driver (Camofox or Playwright). Unified JSON schema across all backends; machine-readable error_code for agent branching. Field reports and agent-use questions: Agent Waystation #22 Teahouse: https://github.com/ythx-101/openclaw-qa/discussions/22

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/ythx-101--x-tweet-fetcher/badge.svg)](/listings/ythx-101--x-tweet-fetcher)

HTML

<a href="/listings/ythx-101--x-tweet-fetcher"><img src="/api/listings/ythx-101--x-tweet-fetcher/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/ythx-101/x-tweet-fetcher
cd x-tweet-fetcher && pip install .

# Or in one step (not published on PyPI):
pip install git+https://github.com/ythx-101/x-tweet-fetcher

# Single tweet — works instantly, zero configuration
xtf --url https://x.com/user/status/1234567890

# User timeline (needs a Nitter instance, see below)
export XTF_NITTER=http://127.0.0.1:8788
xtf --user elonmusk --limit 20

# Search
xtf --search "openclaw" --limit 10

# Human-readable output instead of JSON
xtf --user elonmusk --text-only

Prefer not to install? `python3 scripts/fetch_tweet.py --url ...` works straight from the clone (same flags).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 106 lines
  1. ---
  2. name: x-tweet-fetcher
  3. description: >
  4. Fetch tweets, replies, timelines, search results, X Lists, and X Articles
  5. from X/Twitter without login or API keys. Single tweets: zero dependencies
  6. (FxTwitter). Timelines/search/replies: a Nitter instance (XTF_NITTER).
  7. Lists: Nitter first, browser fallback. Articles: a browser driver
  8. (Camofox or Playwright). Unified JSON
  9. schema across all backends; machine-readable error_code for agent branching.
  10. Field reports and agent-use questions: Agent Waystation #22 Teahouse:
  11. https://github.com/ythx-101/openclaw-qa/discussions/22
  12. ---
  13. # X Tweet Fetcher
  14. Fetch tweets from X/Twitter without authentication. For agent-use stories, failures, and field reports, start from **[#22 Teahouse / 茶座](https://github.com/ythx-101/openclaw-qa/discussions/22)**.
  15. ## Feature Overview
  16. | Feature | Command | Dependencies |
  17. |---------|---------|-------------|
  18. | Single tweet | `xtf --url <tweet_url>` | None (zero deps) |
  19. | Reply threads | `xtf --url <tweet_url> --replies` | Nitter or browser |
  20. | User timeline | `xtf --user <username> --limit 50` | Nitter or browser |
  21. | Search | `xtf --search "<query>"` | Nitter |
  22. | User profile | `xtf --user-info <username>` | None (zero deps) |
  23. | X List | `xtf --list <list_url_or_id>` | Nitter, browser fallback (Camofox/Playwright) |
  24. | X Article | `xtf --article <url_or_id>` | Browser (Camofox/Playwright) |
  25. | Mentions monitor | `xtf --monitor @<username>` | Nitter or browser |
  26. `python3 scripts/fetch_tweet.py` accepts the same flags (v1-compatible entry point).
  27. ## Basic Usage (Zero Dependencies)
  28. ```bash
  29. # JSON output (default)
  30. xtf --url https://x.com/user/status/1234567890
  31. # Human-readable
  32. xtf --url https://x.com/user/status/1234567890 --text-only
  33. # Output covers: text, author, stats (likes/retweets/views), media URLs,
  34. # quoted tweets, and full article text for tweet-embedded articles.
  35. ```
  36. ## Timeline / Search / Replies (Nitter)
  37. ```bash
  38. export XTF_NITTER=http://127.0.0.1:8788 # your instance; comma-separate for failover
  39. xtf --user elonmusk --limit 20
  40. xtf --search "openclaw" --limit 10
  41. xtf --url https://x.com/user/status/123 --replies
  42. ```
  43. Backend selection: `--backend auto` (default, Nitter first then browser), `--backend nitter`, `--backend browser`.
  44. ## Lists & Articles
  45. Lists try Nitter first (plain HTTP) and fall back to the browser; Articles need the browser.
  46. ```bash
  47. # Articles (and the Lists browser fallback) need a browser driver:
  48. # Camofox running on localhost:9377 (default), or:
  49. export XTF_BROWSER=playwright
  50. xtf --list https://x.com/i/lists/1455045069516357634 --limit 30
  51. xtf --article https://x.com/i/article/2011779830157557760
  52. ```
  53. Note: full X Article text requires X login; without it you get title + public preview (`is_partial: true`).
  54. ## Mentions Monitor (cron-friendly)
  55. ```bash
  56. xtf --monitor @yourhandle
  57. # exit 0 = no new mentions, 1 = new mentions found, 2 = setup error
  58. # First run builds a baseline silently; later runs report only new URLs.
  59. ```
  60. ## Error Handling for Agents
  61. Every error result includes `error` (message) and `error_code`:
  62. `invalid_input` · `not_found` · `rate_limited` · `upstream_down` · `backend_unavailable` · `all_backends_failed` (with per-backend `error_causes`).
  63. ## Python API
  64. ```python
  65. from xtf import Router
  66. router = Router()
  67. tweets = router.fetch_timeline("elonmusk", limit=20) # list[Tweet]
  68. print(tweets[0].to_dict())
  69. ```
  70. ## Directory Structure
  71. ```
  72. src/xtf/
  73. ├── backends/ # fxtwitter.py, nitter.py, browser.py
  74. ├── parsers/ # pure parsing functions, fixture-tested
  75. ├── router.py # auto-fallback
  76. ├── monitor.py # mentions monitor
  77. └── cli.py # the `xtf` command
  78. scripts/fetch_tweet.py # v1-compatible shim
  79. ```
  80. Looking for Chinese-platform fetching (Weibo/Bilibili/WeChat) or tweet growth tracking? Those moved out of this repo in v2 — see MIGRATION.md.
README.md · README · 279 lines
  1. <div align="center">
  2. # x-tweet-fetcher
  3. **Fetch X/Twitter tweets, replies, timelines, lists, and articles — no login, no API keys.**
  4. [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
  5. [![Python 3.10+](https://img.shields.io/badge/Python-3.10+-green.svg)](https://www.python.org)
  6. [![GitHub stars](https://img.shields.io/github/stars/ythx-101/x-tweet-fetcher?style=social)](https://github.com/ythx-101/x-tweet-fetcher)
  7. *Three backends · Auto fallback · Unified JSON schema · Built for AI agents*
  8. [Quick Start](#-quick-start) · [Backends](#-three-backends) · [Capabilities](#-capabilities) · [When to use](#-when-to-use--alternatives) · [FAQ](#-faq) · [Python API](#-python-api) · [Self-hosted Nitter](#-self-hosted-nitter) · [Migrating from v1](#-migrating-from-v1)
  9. </div>
  10. ---
  11. **x-tweet-fetcher (`xtf`) is a Python CLI and library that lets AI agents and developers fetch X/Twitter tweets, replies, timelines, search results, Lists and X Articles without logging in or using the X API — it routes between FxTwitter, Nitter and a browser driver with automatic fallback and returns one unified JSON schema.**
  12. ### 中文简介
  13. 免登录、免 API key 抓取 X(推特)的推文、回复、时间线、搜索、Lists 和长文(X Articles)。适合给 AI agent(OpenClaw、Claude Code 等能跑命令行的 agent)当“读推特”的工具,无需 API,输出统一 JSON,并可存入本地 SQLite 推文库。单条推文零依赖;时间线/搜索需要 Nitter;Lists 可走 Nitter 或浏览器;X Article 需要浏览器。
  14. ---
  15. ## 😤 Problem
  16. ```
  17. You: fetch that tweet / list / article for me
  18. AI: I can't access X/Twitter. Please copy-paste the content manually.
  19. You: ...seriously?
  20. ```
  21. X has no free API. Scraping gets you blocked. Browser automation is fragile in headless environments.
  22. **x-tweet-fetcher** solves this with **smart backend routing**: FxTwitter for single tweets (zero deps), Nitter for timelines and search (direct HTTP), a browser driver for everything else — with automatic fallback between them.
  23. ## 🚀 Quick Start
  24. ```bash
  25. git clone https://github.com/ythx-101/x-tweet-fetcher
  26. cd x-tweet-fetcher && pip install .
  27. # Or in one step (not published on PyPI):
  28. pip install git+https://github.com/ythx-101/x-tweet-fetcher
  29. # Single tweet — works instantly, zero configuration
  30. xtf --url https://x.com/user/status/1234567890
  31. # User timeline (needs a Nitter instance, see below)
  32. export XTF_NITTER=http://127.0.0.1:8788
  33. xtf --user elonmusk --limit 20
  34. # Search
  35. xtf --search "openclaw" --limit 10
  36. # Human-readable output instead of JSON
  37. xtf --user elonmusk --text-only
  38. ```
  39. Prefer not to install? `python3 scripts/fetch_tweet.py --url ...` works straight from the clone (same flags).
  40. ## 🔀 Three Backends
  41. | Backend | Deps | Speed | Covers |
  42. |---------|------|-------|--------|
  43. | **fxtwitter** | None (stdlib) | ⚡⚡ | Single tweets, user profiles |
  44. | **nitter** | A Nitter instance | ⚡ | Timeline, search, replies, mentions, **Lists** |
  45. | **browser** | Camofox *or* Playwright | 🐢 | Everything above + **X Articles** (also the Lists fallback) |
  46. | **auto** (default) | Best available | ⚡→🐢 | Nitter first, browser fallback |
  47. ```bash
  48. xtf --user elonmusk # auto (default)
  49. xtf --user elonmusk --backend nitter # direct HTTP only
  50. xtf --list 1455045069516357634 # Lists: Nitter first, browser fallback
  51. ```
  52. **Browser driver** defaults to Camofox (`localhost:9377`). Playwright users:
  53. ```bash
  54. pip install ".[playwright]" # from the clone
  55. export XTF_BROWSER=playwright # or: --browser-driver playwright
  56. ```
  57. ## 📊 Capabilities
  58. | Feature | Flag | Backend |
  59. |---------|------|---------|
  60. | Single tweet (text, stats, media, quotes) | `--url` | fxtwitter |
  61. | Reply comments (threaded) | `--url --replies` | nitter / browser |
  62. | User timeline (paginated) | `--user` | nitter / browser |
  63. | Search | `--search` | nitter |
  64. | User profile | `--user-info` | fxtwitter → nitter |
  65. | X List tweets | `--list` | nitter / browser |
  66. | X Article full text | `--article` | browser |
  67. | Mentions monitor (incremental, cron-friendly) | `--monitor` | nitter / browser |
  68. | Archive fetch results (dedupe, SQLite) | `--ledger <db>` | any |
  69. | Search / stats the archive (offline) | `--ledger <db> --query/--stats` | offline |
  70. **Exit codes** (cron-friendly): `0` success / no new mentions · `1` error / new mentions found · `2` monitor setup error.
  71. **Errors are machine-readable.** Every failure carries `error` (human message) plus `error_code` — one of `invalid_input`, `not_found`, `rate_limited`, `upstream_down`, `backend_unavailable`, `all_backends_failed` — so agents can branch on it. `all_backends_failed` additionally includes per-backend `error_causes`.
  72. ## 🧭 When to use / Alternatives
  73. | Approach | Login / keys | Good for | Trade-off |
  74. |---|---|---|---|
  75. | Official X API | Developer account + API keys | Production, write access, guaranteed terms | Account setup, paid access tiers |
  76. | Account-based scrapers | Requires X accounts / cookies | Large-scale collection | Account ban risk, credential handling |
  77. | **x-tweet-fetcher** | None for single tweets; your own Nitter / browser for the rest | AI agents reading tweets on demand, small archives | Depends on FxTwitter / Nitter / page structure; read-only |
  78. **Use it when…**
  79. - an AI agent or script needs to read a tweet, thread, timeline or List on demand, without handling X credentials;
  80. - you want one JSON shape no matter which backend answered;
  81. - you want a small local archive of what you fetched (`--ledger`).
  82. **Don't use it when…**
  83. - you need to post or do anything other than read — xtf is read-only;
  84. - you need contractual guarantees or official support for production workloads — use the official X API.
  85. ## 📚 推文库 (Ledger)
  86. `--ledger <db>` turns xtf into a **fetch + archive + query** local tweet library: every timeline / search / list / replies / single-tweet fetch is archived into a SQLite DB, deduped by `tweet_id` (`INSERT OR IGNORE`, idempotent). Schema is compatible with the tweet-ledger (OpenClaw) `tweets` table, so the same DB can be read by both tools.
  87. ```bash
  88. # Fetch + archive a timeline
  89. xtf --user YuLin807 --limit 20 --ledger ~/tweets.db
  90. # Search the archive (offline)
  91. xtf --ledger ~/tweets.db --query "sop"
  92. # Stats: totals, languages, media/urls, time ranges
  93. xtf --ledger ~/tweets.db --stats
  94. ```
  95. Behavior without `--ledger` is unchanged (3.0.0-compatible). Archiving never breaks a successful fetch — on failure the JSON envelope carries `ledger_error` instead. Single-tweet (fxtwitter) dicts lack `tweet_id`, so the CLI injects it from the URL; `--replies` results are archived with `is_reply=1` and `in_reply_to_status_id` pointing at the parent tweet.
  96. `tweets` table: `tweet_id` (PK) · `created_at` · `full_text` · `lang` · `source_file` · `is_reply` · `in_reply_to_status_id` · `retweeted_status_id` · `quoted_status_id` · `urls_json` · `media_json` · `raw_json` · `imported_at`
  97. End-to-end integration test record: [docs/e2e-integration.md](docs/e2e-integration.md).
  98. ## 🐍 Python API
  99. ```python
  100. from xtf import Router, NotFound, RateLimited
  101. router = Router() # backend="auto"
  102. tweet = router.fetch_tweet("user", "1234567890") # dict, v1-compatible shape
  103. tweets = router.fetch_timeline("user", limit=20) # list[Tweet]
  104. replies = router.fetch_replies("user", "1234567890")
  105. results = router.search("openclaw", limit=10)
  106. for tw in tweets:
  107. print(tw.author, tw.likes, tw.text)
  108. print(tw.to_dict()) # JSON-ready
  109. ```
  110. All backends normalize into one `Tweet` / `Reply` / `Profile` / `Article` schema — your downstream prompt only ever needs to describe one shape.
  111. ## ⚙️ Configuration
  112. Everything is an environment variable (CLI flags override):
  113. | Variable | Default | Meaning |
  114. |----------|---------|---------|
  115. | `XTF_NITTER` | `http://127.0.0.1:8788` | Comma-separated Nitter instances, tried in order with failover |
  116. | `XTF_BROWSER` | `camofox` | Browser driver: `camofox` or `playwright` |
  117. | `XTF_BROWSER_PORT` | `9377` | Camofox HTTP port |
  118. | `XTF_LANG` | `zh` | Message language: `zh` or `en` |
  119. | `XTF_CACHE_DIR` | `~/.x-tweet-fetcher` | Mentions-monitor cache |
  120. `NITTER_URL` (the v1 name) is still honored as a fallback for `XTF_NITTER`.
  121. ## 🏗 Self-hosted Nitter
  122. Public Nitter instances are unreliable and frequently dead. **Self-hosting is strongly recommended** for timeline/search/replies:
  123. Running your own instance needs two things: a `nitter.conf` (copy the official `nitter.example.conf` and edit it) and a separately running Redis/Valkey. Follow the official guide, including the docker-compose option: <https://github.com/zedeus/nitter#docker>. The official single-container example is:
  124. ```bash
  125. docker run -v $(pwd)/nitter.conf:/src/nitter.conf -d --network host zedeus/nitter:latest
  126. ```
  127. Start Redis first, and set the port in `nitter.conf` to the one you want. Then point xtf at it:
  128. ```bash
  129. export XTF_NITTER=http://127.0.0.1:<port>
  130. ```
  131. Multiple instances failover automatically:
  132. ```bash
  133. export XTF_NITTER=http://127.0.0.1:8788,https://your-backup-instance.example
  134. ```
  135. If no instance is reachable, you get a clear error (`error_code: "all_backends_failed"`, with each backend's reason — e.g. `backend_unavailable` — under `error_causes`) telling you exactly what to set. Never a silent empty result.
  136. ## 📁 Project Structure
  137. ```
  138. src/xtf/
  139. ├── models.py # Tweet / Reply / Profile / Article dataclasses
  140. ├── backends/
  141. │ ├── fxtwitter.py # single tweets + profiles
  142. │ ├── nitter.py # direct HTTP, multi-instance failover
  143. │ └── browser.py # Camofox / Playwright snapshot fetching
  144. ├── parsers/ # pure functions, locked by fixture tests
  145. ├── router.py # auto-fallback chain
  146. ├── monitor.py # incremental mentions monitor
  147. └── cli.py # the `xtf` command
  148. scripts/fetch_tweet.py # v1-compatible entry point (thin shim)
  149. tests/fixtures/ # captured page structures — regression protection
  150. ```
  151. ## ❓ FAQ
  152. **Do I need an X/Twitter API key or account? / 需要 API key 或登录吗?**
  153. No. Single tweets go through FxTwitter with no setup. Full X Article text needs a browser that is already logged in to X; otherwise you get the title plus the public preview (`is_partial: true`).
  154. **Why does `--user` / `--search` fail with `all_backends_failed`?**
  155. No Nitter instance was reachable. Set `XTF_NITTER` to a working instance and check `error_causes` in the output for each backend's reason.
  156. **Can I use public Nitter instances?**
  157. Yes — `XTF_NITTER` accepts several comma-separated instances and fails over between them. Public instances are often unreliable, so [self-hosting](#-self-hosted-nitter) is recommended.
  158. **How do I use it from an AI agent (OpenClaw, Claude Code, others)?**
  159. Have the agent run `xtf ...`, parse the JSON, and branch on `error_code`. The repo ships a [SKILL.md](SKILL.md) describing the commands.
  160. **Can it fetch X Lists and X Articles?**
  161. Lists: Nitter first, browser fallback. Articles: browser only (Camofox or Playwright).
  162. **How do I keep a local archive? / 怎么存档?**
  163. Add `--ledger <db>` to any fetch; query it offline with `--ledger <db> --query <term>` or `--stats`.
  164. **Is it on PyPI?**
  165. Not currently. Install from source: `pip install git+https://github.com/ythx-101/x-tweet-fetcher`.
  166. **Does it post or write to X?**
  167. No, it is read-only.
  168. ## 🔄 Migrating from v1
  169. `python3 scripts/fetch_tweet.py` still works with all v1 flags and exit codes, and JSON fields are unchanged for every mode **except `--search`**, whose per-tweet schema is now unified with `--user` (fields renamed, `url`/`has_media`/`media_urls` dropped). See [MIGRATION.md](MIGRATION.md) for the full list, including where the analytics/China/Obsidian scripts went (spoiler: their own repos — this project is now purely about fetching tweets; the old world lives at the `v1-legacy` tag).
  170. ## 🧪 Development
  171. ```bash
  172. pip install -e ".[dev]"
  173. pytest # all parsers locked by fixture tests
  174. ruff check src tests
  175. ```
  176. When Nitter or X change their page structure, capture a fresh snapshot into `tests/fixtures/` — the failing test will show exactly which parser and field broke.
  177. ## 🙏 Acknowledgments
  178. - **[Nitter](https://github.com/zedeus/nitter)** by [zedeus](https://github.com/zedeus) — self-hosted Twitter frontend
  179. - **[FxTwitter](https://github.com/FxEmbed/FxEmbed)** — public API for single tweet data
  180. - **[Camofox (camofox-browser)](https://github.com/jo-inc/camofox-browser)** — anti-fingerprint browser, default browser driver
  181. - **[Playwright](https://github.com/microsoft/playwright)** — alternative browser automation driver
  182. - **[OpenClaw](https://github.com/openclaw/openclaw)** — AI agent framework this tool grew up in
  183. ## 📜 License
  184. [MIT](LICENSE)
  185. ---
  186. <div align="center">
  187. *Three backends. Auto fallback. Built for AI agents.*
  188. **[GitHub](https://github.com/ythx-101/x-tweet-fetcher)** · **[Issues](https://github.com/ythx-101/x-tweet-fetcher/issues)** · **[#22 Teahouse](https://github.com/ythx-101/openclaw-qa/discussions/22)** · **[Agent Waystation](https://github.com/ythx-101/openclaw-qa)**
  189. </div>

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.