Skip to content
Back to search

by2kb

Forward videos from IM to transcript, Markdown, and your knowledge base — with raw and skill-updated outputs.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/charlesmpc--by2kb/badge.svg)](/listings/charlesmpc--by2kb)

HTML

<a href="/listings/charlesmpc--by2kb"><img src="/api/listings/charlesmpc--by2kb/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install the published package with local Whisper and YouTube support:

pipx install "by2kb[asr-whisper,youtube]"
by2kb init --preset agent-local
by2kb models install

Cloud Doubao ASR remains available for machines that should not run Whisper locally:

pipx install "by2kb[asr-doubao,youtube]"
by2kb init

`by2kb init` guides you through four decisions:

1. where the local knowledge base should live; 2. whether transcription uses local faster-whisper or cloud Doubao AUC; 3. local model settings, or the private TOS bucket and Doubao credentials; 4. whether summaries run in an agent (`agent`), through a standalone LLM API (`api`), or remain disabled.

After installing the selected optional ASR runtime/model, verify the deployment:

by2kb doctor
by2kb doctor --json

Bilibili and YouTube are enabled by default, and yt-dlp ships with the base package. YouTube uses available captions first, then the configured ASR when audio is needed. No separate platform configuration is required. Explicit source overrides remain respected; see [upgrading source defaults](docs/upgrading.md#bilibili-and-youtube-defaults) if an older configuration deliberately or historically lists only Bilibili.

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 749 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. # by2kb
  2. **Forward a video. Keep the knowledge.**
  3. `by2kb` turns a video URL into three durable Markdown artifacts in your own knowledge
  4. base: the evidence-preserving transcript, a short “is this worth reading?” abstract,
  5. and long-form study notes.
  6. The primary user is an **agent user**. Send a Bilibili URL or media attachment to an
  7. agent such as Hermes; `by2kb` handles media retrieval and the selected local or cloud
  8. ASR provider, while the agent uses its existing model authentication to produce both summaries.
  9. Standalone users can run the same pipeline with their own OpenAI-compatible API key.
  10. **Learning-topic discovery (new in 0.7.0).** Send
  11. `by2kb 我想了解一下 TiDB 向量检索` through the updated Hermes plugin to get
  12. separate Bilibili and YouTube recommendation lists (up to 3–5 videos per enabled
  13. source in 0.7.5), then reply with a globally unique number to enter the normal
  14. transcription and knowledge-base workflow. Search sources are individually configurable;
  15. caption-only previews never download audio or run ASR. See
  16. [topic search and configuration](docs/topic-search.md) for CLI usage, selection
  17. state, preview limits, and deployment requirements. Current built-in search sources
  18. are Bilibili and YouTube; podcast and article ingestion remain future extensions.
  19. ## See it in action
  20. **Save a video you like. Or explore a topic you want to learn.**
  21. - **Save a video:** send its link to Hermes, then keep the transcript, short abstract,
  22. and study notes in your own knowledge base.
  23. - **Explore a topic:** ask Hermes about a learning topic, browse relevant video
  24. recommendations, and reply with a number to transcribe and save your selection.
  25. https://github.com/user-attachments/assets/0cbec29a-f2a7-4279-bff1-a3c021574808
  26. *36.6-second demo (by2kb 0.7.3): two Hermes-on-Telegram workflows, with Chinese
  27. captions. Processing waits have been cut for brevity, not shown in real time.
  28. Sound optional.*
  29. > **[Latest release](https://github.com/Charlesmpc/by2kb/releases/latest).** Agent-guided installation, a cloud-free local Whisper
  30. > default, upgrade-safe personalization, Bilibili/YouTube ingestion, Agent/API
  31. > enrichment, task control, and the Hermes plugin are implemented. A resident service,
  32. > native Telegram/Lark bots, and remote knowledge-base sinks remain planned.
  33. ## Start here
  34. Requires Python 3.12+, `pipx`, and `ffmpeg`/`ffprobe` on PATH. YouTube player
  35. challenges also need Deno 2.3+ or Node.js 22+; matching EJS scripts ship with the
  36. Python dependencies. `by2kb doctor` checks both. See [YouTube JS setup](docs/source-providers.md#youtube-javascript-support).
  37. ### Ask your Agent to install it
  38. Send this prompt to an Agent with terminal access:
  39. ```text
  40. Read and follow the official by2kb installation skill:
  41. https://raw.githubusercontent.com/Charlesmpc/by2kb/main/skills/install-by2kb/SKILL.md
  42. ```
  43. The installation Skill defaults to local faster-whisper, Agent-hosted summaries,
  44. Bilibili and YouTube sources, and a local Markdown knowledge base. It does not clone
  45. the repository or ask for cloud credentials in chat.
  46. Upgrades replace application and adapter code but preserve `$BY2KB_HOME` (normally
  47. `~/.by2kb`), the configured knowledge-base folder, downloaded models, job history,
  48. credentials, and custom Skills. See [Upgrading](docs/upgrading.md).
  49. ### Install it yourself
  50. Install the published package with local Whisper and YouTube support:
  51. ```bash
  52. pipx install "by2kb[asr-whisper,youtube]"
  53. by2kb init --preset agent-local
  54. by2kb models install
  55. ```
  56. Cloud Doubao ASR remains available for machines that should not run Whisper locally:
  57. ```bash
  58. pipx install "by2kb[asr-doubao,youtube]"
  59. by2kb init
  60. ```
  61. `by2kb init` guides you through four decisions:
  62. 1. where the local knowledge base should live;
  63. 2. whether transcription uses local faster-whisper or cloud Doubao AUC;
  64. 3. local model settings, or the private TOS bucket and Doubao credentials;
  65. 4. whether summaries run in an agent (`agent`), through a standalone LLM API (`api`),
  66. or remain disabled.
  67. After installing the selected optional ASR runtime/model, verify the deployment:
  68. ```bash
  69. by2kb doctor
  70. by2kb doctor --json
  71. ```
  72. Bilibili and YouTube are enabled by default, and yt-dlp ships with the base package.
  73. YouTube uses available captions first, then the configured ASR when audio is needed.
  74. No separate platform configuration is required. Explicit source overrides remain
  75. respected; see [upgrading source defaults](docs/upgrading.md#bilibili-and-youtube-defaults)
  76. if an older configuration deliberately or historically lists only Bilibili.
  77. ### Agent-first: Hermes
  78. After selecting `agent` during initialization:
  79. ```bash
  80. by2kb agent install hermes
  81. hermes gateway restart
  82. ```
  83. These commands are for the legacy copy-based adapter installation. If Hermes
  84. already manages your by2kb plugin through Git or the catalog, do not reinstall it
  85. with the copy installer. See [native plugin setup](docs/agent-integration.md#native-hermes-installation-and-first-setup)
  86. and [upgrade ownership](docs/upgrading.md#managed-code). Catalog submission is pending.
  87. Now an authorized user can send a Bilibili, `b23.tv`, YouTube, or `youtu.be` URL to
  88. the Telegram Hermes bot.
  89. The plugin acknowledges immediately, runs transcription in the background, calls the
  90. Hermes host model through bounded staged operations, writes all three artifacts, and
  91. replies with the short abstract and knowledge-base paths. No MCP server and no
  92. separate by2kb LLM key are required.
  93. ### Standalone
  94. Select `api` during initialization, provide an OpenAI-compatible endpoint, model and
  95. API key, then run:
  96. ```bash
  97. by2kb ingest "https://www.bilibili.com/video/<bvid>/"
  98. by2kb ingest ./meeting.mp3
  99. by2kb ingest ./lecture.mp4
  100. ```
  101. See [Agent integration](docs/agent-integration.md) for the reusable staged
  102. `ingest → next → submit` enrichment contract used by other agent hosts.
  103. See [Agent task control](docs/task-control.md) for the versioned
  104. `status`/`wait`/`cancel`/`retry` protocol and checkpoint-aware retries.
  105. See [ASR providers](docs/asr-providers.md) for optional local faster-whisper setup,
  106. explicit model installation, and the existing cloud Doubao path.
  107. **Bilibili HTTP 412?** Version 0.5.0 can recover from a blocked `view` request using
  108. `pagelist` to obtain CID and continue native audio acquisition—no browser required.
  109. An opt-in browser Plan B uses a dedicated Chromium session when native acquisition
  110. fails. It requires the `browser` extra and manual login when the site requests it;
  111. neither route guarantees recovery from all site risk controls. See the
  112. [browser setup and fallback guide](docs/source-providers.md#browser-fallback-043-opt-in).
  113. See [Source providers](docs/source-providers.md) for configurable provider priority,
  114. default yt-dlp routing, caption policy, audio fallback, and cookie handling.
  115. See [Local media](docs/local-media.md) for supported formats, content-addressed
  116. deduplication, ffmpeg requirements, and attachment handling.
  117. See [Diagnostics](docs/doctor.md) for every read-only check and the Agent-facing JSON schema.
  118. See [Transcript quality](docs/transcript-quality.md) for deterministic enrichment gates,
  119. recorded metrics, and warning behavior.
  120. See [Long-form enrichment](docs/long-form-enrichment.md) for segment-safe planning,
  121. intermediate caching, recursive reduction, and provenance.
  122. See [Agent enrichment provider](docs/agent-enrichment-provider.md) for the bounded
  123. callback protocol and API-key versus subscription/OAuth authentication tradeoffs.
  124. The name can be read as **B/Y to KB** — Bilibili and YouTube to Knowledge Base — while
  125. the architecture is intended to support more video sources over time.
  126. ## Why
  127. Most video tools optimize for watching, summarizing, or taking notes inside a browser.
  128. That is useful, but it is not the workflow this project wants:
  129. - Interesting videos are often discovered in a mobile app.
  130. - A browser extension should not be required to keep a desktop computer online.
  131. - The transcript should become a portable artifact, not remain in extension storage.
  132. - The original transcript and AI-edited output should be kept separately.
  133. - Summarization should follow the user's own methods, prompts, and domain knowledge.
  134. - Markdown and source JSON should remain the source of truth; a vector database is only
  135. an optional index.
  136. There is also an obvious alternative that this project deliberately does not take as
  137. the primary path: sending the video itself to a multimodal bot. In practice it hits
  138. two walls:
  139. - **Access.** Bots are usually locked out of the watch page — platforms gate playback
  140. behind logins and anti-scraping, and generally only allow metadata-level access. The
  141. same gating is exactly why subtitle and metadata APIs remain the most reliable entry
  142. point.
  143. - **Cost.** Even when the content is reachable, feeding video into a model consumes
  144. orders of magnitude more tokens than working with text.
  145. So `by2kb` works down a cost ladder, from lightest to heaviest:
  146. 1. **Native transcript** — if subtitles exist, the job is nearly free: metadata-level
  147. access is enough and the payload is small text.
  148. 2. **Audio stream + ASR** — when there are no subtitles, retrieve the audio track and
  149. transcribe it.
  150. 3. **Full media download** — when audio and video are not delivered separately,
  151. download the original media file (e.g. an mp4) and extract the audio.
  152. Each step down costs more bandwidth, compute, and tokens, so the service always tries
  153. the lightest step first. And at every step it captures content, never the video
  154. itself: the knowledge-base entry is a cheap, durable gist — transcript or ASR text
  155. plus a link back — so the platform keeps bearing hosting and anti-scraping cost,
  156. while a topic worth rewatching stays one click away from the original.
  157. `by2kb` treats a video link as an asynchronous knowledge-ingestion job.
  158. ## Agent-first experience
  159. ```text
  160. Bilibili mobile app
  161. │
  162. │ Share video URL
  163. ▼
  164. Telegram Hermes bot
  165. │
  166. │ Accepted: queued
  167. ▼
  168. Hermes plugin + by2kb CLI
  169. │
  170. ├─ resolve metadata
  171. ├─ retrieve audio
  172. ├─ run local Whisper or a configured cloud ASR provider
  173. ├─ normalize transcript
  174. ├─ save raw Markdown + source JSON
  175. ├─ create a short abstract
  176. └─ create long-form study notes
  177. │
  178. ▼
  179. Local filesystem / Obsidian vault
  180. │
  181. └─ IM notification with links and status
  182. ```
  183. From the user's point of view:
  184. 1. Find an interesting video.
  185. 2. Tap **Share** and send it to the configured IM bot.
  186. 3. Receive an immediate acknowledgement and job status.
  187. 4. Receive three knowledge artifacts when processing finishes:
  188. - **Raw** — minimally normalized transcript with source timestamps.
  189. - **Abstract** — a one-minute interest check: what the video argues, what the reader
  190. will learn, and whether it is worth going deeper.
  191. - **Updated / study notes** — a long, structured learning guide with a knowledge
  192. map, guided walkthrough, claims and evidence, concepts, questions, and actions.
  193. Agent hosts can expose progress and lifecycle controls without parsing logs:
  194. ```bash
  195. by2kb status <job-id> --json
  196. by2kb wait <job-id> --timeout 30 --json
  197. by2kb cancel <job-id> --json
  198. by2kb retry <job-id> --json
  199. ```
  200. ## Output model
  201. Every accepted video gets a stable content directory. JSON artifacts use fixed
  202. names; Markdown artifacts are named `raw.<Title>.md`, `short.<Title>.md`, and
  203. `long.<Title>.md`. The parent directory already carries the stable video ID, while
  204. the prefix makes reading depth obvious in ordinary file listings (naming contract:
  205. `docs/tech-design-m1.md` §3.4):
  206. ```text
  207. library/
  208. youtube/<video-id>/
  209. source.json
  210. transcript.json
  211. raw.<Title>.md
  212. short.<Title>.md
  213. long.<Title>.md
  214. bilibili/<bvid>/
  215. source.json
  216. transcript.json
  217. raw.<Title>.md
  218. short.<Title>.md
  219. long.<Title>.md
  220. ```
  221. ### `raw.<Title>.md`
  222. The raw document should be deterministic and reproducible. It contains:
  223. - source URL and canonical video ID;
  224. - title, author/channel, duration, language, and capture time;
  225. - transcript provider and whether the track is human- or machine-generated;
  226. - timestamped transcript segments linking back to the original video;
  227. - no invented facts or silent rewriting.
  228. ### `short.<Title>.md`
  229. The abstract is deliberately short and decision-oriented. It contains a one-sentence
  230. summary, two to four concrete learning outcomes, and a recommendation describing who
  231. will benefit from reading the study notes or watching the source.
  232. ### `long.<Title>.md` (deep study notes)
  233. The long document is generated from `raw.<Title>.md` and records:
  234. - skills and versions used;
  235. - model/provider metadata where applicable;
  236. - core thesis, knowledge map, guided walkthrough, claims and evidence, concepts,
  237. questions to verify, and a study/action list;
  238. - links back to the raw artifact and original video;
  239. - processing timestamp, so it can be regenerated when skills improve.
  240. Keeping generated files separate from `raw.<Title>.md` prevents an AI rewrite from replacing
  241. the evidence.
  242. ### LLM access and authentication
  243. Summary generation does require an LLM, but it does **not** require a browser login.
  244. There are two execution paths:
  245. - Agent users select `external_agent`. The Hermes reference plugin borrows the active
  246. Hermes provider/model through its host-owned LLM API; `by2kb` never receives that
  247. credential and no nested agent loop is created.
  248. - Standalone deployments select `api`. `by2kb` calls an OpenAI-compatible endpoint
  249. with a server-side API key and never stores that key in an artifact. The default
  250. endpoint preset is Volcengine Ark. To use OpenAI, set:
  251. ```dotenv
  252. BY2KB_LLM_API_KEY=<your API key>
  253. BY2KB_LLM_MODEL=<an API model available to your project>
  254. BY2KB_LLM_BASE_URL=https://api.openai.com/v1
  255. ```
  256. OpenAI documents API-key authentication in its
  257. [API reference](https://developers.openai.com/api/reference/overview) and its current
  258. text-generation APIs in the
  259. [text generation guide](https://developers.openai.com/api/docs/guides/text).
  260. `by2kb` does not reuse a ChatGPT browser session. Other providers can be used when
  261. they expose the compatible chat-completions endpoint.
  262. ## Personalized skills
  263. A skill is a reusable Markdown-based processing instruction, with optional references,
  264. templates, and scripts. A user can select a default skill set globally and override it
  265. per message or destination.
  266. Example:
  267. ```text
  268. skills/
  269. transcript-cleanup/
  270. SKILL.md
  271. research-video/
  272. SKILL.md
  273. templates/report.md
  274. language-learning/
  275. SKILL.md
  276. ```
  277. Possible skills include:
  278. - punctuation and speech-disfluency cleanup;
  279. - chapter detection with source timestamps;
  280. - concise or detailed summaries;
  281. - extraction of claims, evidence, questions, and action items;
  282. - bilingual notes;
  283. - terminology and entity indexing;
  284. - a team- or domain-specific report format.
  285. Skills transform the updated artifact only. They must not mutate or overwrite the raw
  286. transcript.
  287. ## Transcript providers
  288. Provider adapters hide platform-specific behavior behind a common contract.
  289. ### Phase 1: native transcripts
  290. - **YouTube** — retrieve an existing human or auto-caption track through a configurable
  291. transcript provider. Supadata is one possible provider, not a hard dependency.
  292. - **Bilibili** — dropped 2026-08-17: unauthenticated subtitle tracks proved
  293. unreachable (23/23 video probe) and the authenticated route was rejected, so
  294. Bilibili enters at Phase 2 (audio + ASR) instead. See `docs/tech-design-m1.md`
  295. §7.5 and Appendix A.1.
  296. The service should prefer native transcripts because they are faster, cheaper, and
  297. usually preserve better timestamps.
  298. #### Known-good retrieval paths (from the prior-art projects)
  299. Both inspiration projects are Chrome extensions, but their retrieval logic is plain
  300. HTTP and ports to a server-side service directly:
  301. - **Bilibili** ([bilibili-digest](https://github.com/biuworks/bilibili-digest),
  302. `lib/bili-api.js` + `lib/wbi.js`) — a three-step flow against official web APIs:
  303. 1. `GET https://api.bilibili.com/x/web-interface/view?bvid=...` → `aid` / `cid` /
  304. title / parts (unsigned);
  305. 2. `GET https://api.bilibili.com/x/player/wbi/v2?aid=&cid=&bvid=&wts=&w_rid=` →
  306. subtitle track list; this call requires **WBI signing** (daily-rotating keys from
  307. the `nav` API, a fixed permutation-table mixin key, and an MD5 `w_rid`);
  308. 3. `GET <track.subtitle_url>` (hdslb CDN) → full transcript JSON
  309. `{body: [{from, to, content}]}` in one response, fetched **without** cookies.
  310. Server-side notes: AI subtitle tracks are usually empty for logged-out sessions —
  311. the extension works only because it inherits the user's own browser session
  312. (`credentials: "include"` on `api.bilibili.com`), an option a server-side service
  313. does not legitimately have. The `SESSDATA` route was evaluated and rejected (see
  314. `docs/tech-design-m1.md` §7.5), so `by2kb` does not pursue Bilibili native
  315. subtitles. Business code `-352` means Bilibili risk control blocked the request and
  316. should map to a retryable/backoff state (still relevant to the audio-download
  317. path).
  318. - **YouTube** ([youtube-digest](https://github.com/zarazhangrui/youtube-digest)) —
  319. delegates entirely to the third-party **Supadata** API
  320. (`GET https://api.supadata.ai/v1/transcript?url=...&text=false&mode=native` with an
  321. API key; HTTP 202 returns a job id to poll). This is the simplest path and keeps the
  322. service clear of YouTube's anti-bot surface, at the cost of a paid dependency.
  323. A self-hosted alternative is the
  324. [`youtube-transcript-api`](https://github.com/jdepoix/youtube-transcript-api) Python
  325. library (used by the
  326. [youtube-content skill](https://github.com/NousResearch/hermes-agent/tree/main/skills/media/youtube-content)
  327. in hermes-agent), which talks to YouTube directly; it needs no key but breaks
  328. occasionally when YouTube changes its player internals, so it should sit behind the
  329. same provider interface with its own error taxonomy.
  330. Neither project downloads audio or video streams — they deliberately stay on the
  331. subtitle/metadata surface. Media streams are the fallback for when no native
  332. transcript exists (see Phase 2); how heavily they are gated differs per platform —
  333. `docs/tech-design-m1.md` appendix A records what the Bilibili spike actually reached
  334. unauthenticated.
  335. ### Phase 2: audio fallback
  336. When no usable transcript exists:
  337. 1. obtain the audio through a configurable media provider (where a platform delivers
  338. audio and video muxed rather than separately, the provider downloads the media file
  339. itself — e.g. the original mp4 — and extracts the audio track);
  340. 2. normalize it to an ASR-friendly format;
  341. 3. transcribe it through a configurable ASR provider;
  342. 4. store provenance, timing, model, and confidence metadata;
  343. 5. continue through the same raw/updated pipeline.
  344. Implemented media routes include native Bilibili acquisition and bundled yt-dlp.
  345. ASR supports local faster-whisper by default and optional cloud Doubao AUC. Platform
  346. terms, account security, regional restrictions, cost, and deployment environment
  347. must still be evaluated before use.
  348. A proven Doubao AUC reference flow is included for provider design: local audio is
  349. staged in a private Volcengine TOS bucket, exposed through a 10-minute presigned URL,
  350. submitted and polled asynchronously, and deleted in a `finally` block. Long audio is
  351. split into ordered Opus chunks with bounded concurrency. See
  352. [`docs/reference/doubao-auc-tos-asr.md`](docs/reference/doubao-auc-tos-asr.md) and the
  353. runnable [`examples/doubao_auc_tos_asr.py`](examples/doubao_auc_tos_asr.py).
  354. #### Phase 2b: browser-based source capture
  355. A browser session can sometimes reach a playable stream when plain HTTP fails.
  356. The implemented opt-in Bilibili fallback uses a dedicated Playwright Chromium session:
  357. 1. open the watch URL in a **headless browser**, using the dedicated session's login
  358. state where the site requires it;
  359. 2. extract the media source the page actually plays — from the player object in the
  360. page (e.g. the player response / `playurl` data) or by observing network requests —
  361. and download it (typically a segmented DASH/HLS stream assembled with ffmpeg);
  362. 3. feed the audio into the same ASR pipeline as Phase 2.
  363. This is the same idea as consumer "download the page source" tools — e.g. Xunlei's
  364. new Chrome extension for grabbing page media sources, and hermes-agent's browser
  365. automation skills (stealth browsing via `scrapling`, managed-browser `browser` tools,
  366. and Whisper-based ASR skills) — recast as a headless, server-side adapter.
  367. This path is **off by default** and must stay opt-in per deployment:
  368. - it is the most fragile (player internals change often) and the most
  369. policy-sensitive route — it exists precisely because platforms gate media streams;
  370. - it must respect platform terms, copyright, and the user's own account safety
  371. (a flagged or banned session is a real cost);
  372. - provenance must record that the transcript came from browser capture + ASR, with
  373. lower confidence than native tracks, so downstream skills and readers can tell the
  374. difference.
  375. ## Connectors
  376. ### IM inputs
  377. Implemented:
  378. - Hermes plugin with deterministic Bilibili/YouTube URL interception and Telegram replies;
  379. - Hermes Skill for natural-language/manual requests.
  380. Planned:
  381. - native Telegram and Lark/Feishu bots for users without an agent host;
  382. - generic webhook;
  383. - other agent-host adapters;
  384. - native mobile share target, PWA, or a minimal browser capture extension.
  385. An input adapter should submit a canonical job; it should not contain transcript logic.
  386. ### Knowledge-base outputs
  387. Implemented:
  388. - local/shared filesystem and Obsidian vault directory.
  389. Planned:
  390. - Git-backed Markdown repository;
  391. - Lark/Feishu Wiki or Docx;
  392. - Notion;
  393. - generic webhook/API.
  394. Markdown plus the original transcript JSON is the portable source of truth.
  395. ## CLI reference
  396. `.env` is loaded from `$BY2KB_ENV_FILE`, `<BY2KB_HOME>/.env` (default
  397. `~/.by2kb/.env`), or `./.env`. Artifacts land in
  398. `<library>/<platform>/<video-id>/{source.json,transcript.json,raw.<Title>.md,short.<Title>.md,long.<Title>.md}`;
  399. the abstract and study notes are produced by the configured API or external agent.
  400. Exit codes:
  401. 0 completed, 1 terminal failure, 2 retryable, 3 needs auth, 4 duplicate.
  402. Agent hosts use the durable external-enrichment protocol:
  403. ```bash
  404. by2kb ingest "<video-url>" --enricher external_agent --json
  405. by2kb enrichment next <job-id> \
  406. --provider <provider> --model <model> --json
  407. by2kb enrichment submit <job-id> \
  408. --operation-id <operation-id> --output-file <response.md> \
  409. --provider <provider> --model <model> --json
  410. ```
  411. The agent is not called recursively. `by2kb` leaves durable pending work and supplies
  412. one bounded operation at a time; the host model returns each result until the cached
  413. planner, reducer, and trusted publication path complete both reading depths.
  414. If a video was transcribed before LLM credentials were configured, generate or refresh
  415. only its two summaries without downloading and transcribing the media again:
  416. ```bash
  417. by2kb ingest "https://www.bilibili.com/video/<bvid>/" --re-enrich
  418. ```
  419. Bilibili ingestion goes straight to audio+ASR (native subtitles are not
  420. pursued — `docs/tech-design-m1.md` §7.5); ASR setup details live in
  421. [`docs/reference/doubao-auc-tos-asr.md`](docs/reference/doubao-auc-tos-asr.md).
  422. ## Deployment and integration
  423. The implemented interface is a **local CLI**. `by2kb ingest <url>` resolves the video,
  424. retrieves audio, runs ASR, writes raw artifacts, executes or defers enrichment, and
  425. publishes to the filesystem. Job state, enrichment leases and idempotency live in a
  426. local SQLite store. It runs on a laptop, an agent server, or any host that can spawn a
  427. process.
  428. A resident service and remote client mode are planned for higher-volume deployments;
  429. `BY2KB_SERVER_URL`, HTTP submission, queues and remote workers remain unimplemented.
  430. ### Who needs the service?
  431. - **Users without an agent: eventually.** A standalone IM bot needs a resident webhook
  432. listener, and ingestion is asynchronous (transcript fetch, optional ASR, skill
  433. runs), so the queue, retry, and notification loops must live in a durable resident
  434. process — that is the service. The bot adapter is just another input adapter calling
  435. the service API. ("The bot spawns the CLI per message" works only where the bot
  436. framework can itself execute commands, and gives up queueing, retries, and
  437. concurrency control; it is not the canonical path.)
  438. - **Users with an agent: no.** An agent host already provides the
  439. message surface and the notification channel, so `by2kb` does not need its own bot
  440. identity at all. A small per-agent adapter triggers the CLI. The Hermes reference
  441. integration uses this path today.
  442. ### Agent integration (plugin adapters)
  443. For agent-first users, the adapter of record is a **plugin** on the agent side, not a
  444. `by2kb` bot. The first target is [hermes-agent](https://github.com/NousResearch/hermes-agent):
  445. - a plugin hooking Hermes' `pre_gateway_dispatch` message hook matches a video URL
  446. deterministically, verifies the sender through Hermes authorization, spawns
  447. `by2kb ingest` in a background thread, acknowledges through the agent's own IM
  448. adapter, and returns `skip` so the trigger does not enter a model turn;
  449. - after transcription, the plugin calls Hermes' host-owned LLM for bounded staged
  450. operations (including chunking/reduction when needed), without a nested agent loop
  451. or separate LLM key;
  452. - a companion skill covers the phrased case — "save this video to my KB" — where the
  453. model invokes the CLI through its terminal tool. MCP is not required when both
  454. programs share a host and filesystem.
  455. Other agents follow the same shape: deterministic trigger where the host offers one,
  456. and a skill calling the CLI otherwise. The agreed executor boundary, loop prevention,
  457. Hermes user journey, target CLI, and implementation status are specified in
  458. [`docs/agent-integration.md`](docs/agent-integration.md).
  459. ### Integration matrix
  460. | Scenario | Trigger | Execution | Service needed |
  461. | --- | --- | --- | --- |
  462. | Agent-first (Hermes) | authorized plugin hook on a video URL | CLI subprocess, local mode | No |
  463. | Agent-first, phrased | skill (model judgment) | CLI via the agent's terminal | No |
  464. | No agent | direct CLI today; native IM bot later | local CLI / future service | Not today |
  465. | Scripted / manual | cron, mobile shortcut, hand-typed command | local CLI | No |
  466. The future service will preserve the same job and enrichment contracts when workloads
  467. outgrow a single process.
  468. ## Proposed job contract
  469. ```json
  470. {
  471. "source_url": "https://www.youtube.com/watch?v=...",
  472. "requested_by": "im:user-id",
  473. "destination": "obsidian:videos/ai",
  474. "skills": ["transcript-cleanup", "research-video"],
  475. "options": {
  476. "preferred_languages": ["zh-CN", "en"],
  477. "allow_audio_fallback": false
  478. }
  479. }
  480. ```
  481. A job is idempotent by canonical platform and video ID. Re-sending the same video should
  482. return the existing artifacts unless the user requests a refresh or a different skill
  483. set.
  484. ## Architecture principles
  485. - **Mobile-first capture:** sharing a URL is the primary interface.
  486. - **Asynchronous by default:** IM acknowledgement is immediate; processing continues in
  487. the background.
  488. - **Provider-neutral:** transcript, media, ASR, LLM, IM, and knowledge sinks are adapters.
  489. - **Evidence preservation:** raw and updated outputs are separate and linked.
  490. - **User-owned storage:** portable files are the primary artifacts.
  491. - **Idempotent and retryable:** duplicate messages and transient provider failures are
  492. expected.
  493. - **Observable:** every job records state, provider, timing, error category, and cost when
  494. available.
  495. - **Credential-aware:** platform cookies and API keys are secrets; use least-privilege
  496. accounts and encrypted storage.
  497. - **No browser dependency:** a browser extension may improve capture or authentication,
  498. but the processing service must run headlessly.
  499. See [`docs/architecture.md`](docs/architecture.md) for the proposed components and state
  500. machine.
  501. ## Roadmap
  502. ### Milestone 0 — Product contract
  503. - [x] Define the capture-to-knowledge workflow.
  504. - [x] Separate raw and updated artifacts.
  505. - [x] Define personalized skills as a first-class concept.
  506. - [x] Keep audio retrieval and ASR behind future provider interfaces.
  507. ### Milestone 1 — Local ingestion MVP
  508. - [x] Local SQLite job persistence, status tracking, and deduplication.
  509. - [ ] Remote Job API, queue, and worker service.
  510. - [x] Telegram input through the Hermes plugin.
  511. - [ ] Standalone Telegram input adapter for users without an agent.
  512. - [x] YouTube caption retrieval through the default yt-dlp provider.
  513. - [x] Timestamp-preserving normalization.
  514. - [x] Filesystem/Obsidian Markdown sink (sink contract pinned in
  515. `docs/tech-design-m1.md` §3.7).
  516. - [x] Raw, short-abstract, and long-form study-note generation.
  517. - [x] Packaged default abstract and deep-study skills.
  518. - [x] Durable external-agent enrichment protocol and Hermes reference plugin.
  519. - [x] Guided `by2kb init` for TOS, ASR, enrichment mode, and filesystem output.
  520. - [x] Hermes IM acknowledgement and completion/failure notifications.
  521. ### Milestone 2 — Personalization and more destinations
  522. - [ ] Skill registry, per-user defaults, and per-job overrides.
  523. - [ ] Lark/Feishu input adapter.
  524. - [ ] Lark Wiki/Docx and Notion sinks (projection adapters over the same
  525. artifact set, `docs/tech-design-m1.md` §3.7).
  526. - [x] Regenerate both summary outputs without refetching the transcript (`--re-enrich`).
  527. - [ ] Cost, latency, and provider usage reporting.
  528. ### Milestone 3 — Audio and ASR
  529. - [x] Media retrieval provider interface (contract pinned in
  530. `docs/tech-design-m1.md` §3.8; Bilibili chain spike-verified, Appendix A).
  531. - [x] Bilibili ingestion via this path — its primary route, since unauthenticated
  532. native subtitles were found unreachable and the login route was rejected
  533. (see `docs/tech-design-m1.md` §7.5). Note this makes Bilibili ingestion
  534. non-free: ASR costs either accuracy (self-hosted) or money (hosted).
  535. - [x] Audio extraction and long-audio chunking with ffmpeg.
  536. - [x] Hosted ASR provider interface and Doubao AUC implementation (contract pinned in
  537. `docs/tech-design-m1.md` §3.6; see
  538. `docs/reference/doubao-auc-tos-asr.md`).
  539. - [x] Local faster-whisper ASR provider.
  540. - [ ] Additional ASR providers beyond faster-whisper and Doubao.
  541. - [ ] Segment-level alignment and confidence metadata.
  542. - [ ] Policy and deployment controls per platform/provider.
  543. - [x] Phase 2b: Bilibili browser source capture adapter (opt-in), extracting the
  544. playable media source from the watch page when no native transcript exists,
  545. with `browser_capture + asr` provenance.
  546. ## Non-goals for the first release
  547. - A full video player or browser side panel.
  548. - Real-time subtitle following while a video plays.
  549. - Replacing the user's knowledge-base application.
  550. - Treating generated summaries as a substitute for the source transcript.
  551. - Circumventing access controls or platform restrictions.
  552. ## Prior art
  553. The learning-topic discovery workflow was inspired by
  554. [Agent Reach](https://github.com/Panniantong/Agent-Reach): its platform-routing and
  555. search references demonstrate how an Agent can combine platform-specific discovery
  556. tools (including YouTube `ytsearch` and Bilibili search). by2kb implements its own
  557. configurable search-provider registry, caption-only previews, durable numbered
  558. selection, and ingestion handoff. No Agent Reach source code is copied and Agent
  559. Reach is not a runtime dependency. We thank the project for the inspiration.
  560. `by2kb` was directly inspired by these two projects — they proved the core insight
  561. that a video's native transcript can be retrieved programmatically and turned into a
  562. learning artifact, and `by2kb` exists to move that workflow from a browser extension
  563. into a headless, IM-driven, server-side service:
  564. - [youtube-digest](https://github.com/zarazhangrui/youtube-digest) (MIT, © Zara Zhang) —
  565. a Chrome extension that retrieves YouTube transcripts through Supadata and presents
  566. learning tools (summaries, chapters, translation, notes) in a side panel. It inspired
  567. the raw/updated split, the skill-style prompt templates, and the Supadata provider
  568. path.
  569. - [bilibili-digest](https://github.com/biuworks/bilibili-digest) (MIT) — a Bilibili
  570. adaptation of the same architecture that retrieves official subtitle tracks directly
  571. through Bilibili's web APIs (view → WBI-signed player API → subtitle CDN JSON). It
  572. inspired the Bilibili adapter design, including WBI signing, login-gated AI subtitle
  573. handling, and risk-control (`-352`) error mapping.
  574. `by2kb` is a new server-side, IM-driven ingestion project. No source code from those
  575. projects is included at this stage; when their retrieval logic is ported, their MIT
  576. licenses and attribution requirements (copyright notices, including bilibili-digest's
  577. "portions © Zara Zhang" credit) must be preserved.
  578. ## Contributing
  579. The project is in an early public implementation stage. Discussions and issues about
  580. the following are especially welcome:
  581. - transcript providers and platform reliability;
  582. - the portable skill format;
  583. - knowledge-base destination contracts;
  584. - safe Bilibili authenticated-session handling;
  585. - audio/ASR provider evaluation;
  586. - schemas for timestamped, multilingual transcripts.
  587. Please do not include API keys, cookies, private transcripts, or copyrighted media in
  588. issues or test fixtures.
  589. ## License
  590. MIT — see [LICENSE](LICENSE).

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.