Skip to content
Back to search

meGPT

Code to process many kinds of content by an author into an MCP server

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/adrianco--megpt/badge.svg)](/listings/adrianco--megpt)

HTML

<a href="/listings/adrianco--megpt"><img src="/api/listings/adrianco--megpt/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

git clone https://github.com/adrianco/megpt.git
cd megpt
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Set your API key for LLM enrichment (Q&A extraction, topic synthesis)
export ANTHROPIC_API_KEY=sk-ant-...

# Build the full OKF bundle for an author
python build_okf.py virtual_adrianco --author-display "Adrian Cockcroft"

# Build specific content types
python build_okf.py virtual_adrianco podcast --whisper-model tiny
python build_okf.py virtual_adrianco youtube
python build_okf.py virtual_adrianco story

# Process local file archives (Medium/Blogger posts)
python batch_local_files.py authors/virtual_adrianco/medium_adrianco okf/virtual_adrianco/posts
python batch_local_files.py authors/virtual_adrianco/blogger_perfcap_posts okf/virtual_adrianco/posts

# Generate synthesized topic pages (requires ANTHROPIC_API_KEY)
python okf_synthesizer.py virtual_adrianco

# Start the MCP server (for Claude Desktop / Cursor integration)
python mcp_server_v2.py virtual_adrianco

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 307 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # meGPT - upload an author's content into an LLM
  2. I have 20 years of public content I've produced and presented over my career, and I'd like to have an LLM that is trained to answer questions and generate summaries of my opinions, in my "voice", At this point, I've found a few companies that are building persona's and tried out soopra.ai. To encourage development and competition in this space I have organized my public content and references to sources in this repo.
  3. My own content is stored or linked to in authors/virtual_adrianco and consists of:
  4. - 4 published books (pdf of two provided), ~10 forewords to books, ~100 blog posts (text)
  5. - Twitter archive 2008-2022 (conversation text)
  6. - Mastodon.social - 2021-now https://mastodon.social/@adrianco (RSS at https://mastodon.social/@adrianco.rss)
  7. - Github projects (code)
  8. - Blog posts from https://adrianco.medium.com extracted as text to authors/virtual_adrianco/medium_posts (with extraction script)
  9. - Blog posts from https://perfcap.blogspot.com extracted as text to authors/virtual_adrianco/blogger_percap_posts (with extraction script)
  10. - ~100 presentation decks (images) greatest hits: https://github.com/adrianco/slides/tree/master/Greatest%20Hits
  11. - ~20 podcasts (audio conversations, should be good Q&A training material)
  12. - over 135 videos of talks and interviews (audio/video/YouTube individual videos, playlists, and entire channels)
  13. If another author wants to use this repo as a starting point, clone it and add your own directory of content under authors. If you want to contribute the content freely for other people to use as a training data set, then send a pull request and I'll include it here. The scripts in the code directory are there to help pre-process content for an author by extracting from a twitter or medium archive that has to be downloaded by the account owner.
  14. Creative Commons - attribution share-alike. Permission explicitly granted for anyone to use as a training set to develop the meGPT concept. Free for use by any author/speaker/expert resulting in a Chatbot that can answer questions as if it was the author, with reference to published content. I have called my own build of this virtual_adrianco - with opinions on cloud computing, sustainability, performance tools, microservices, speeding up innovation, Wardley mapping, open source, chaos engineering, resilience, Sun Microsystems, Netflix, AWS etc. etc. I'm happy to share any models that are developed. I don't need to monetize this, I'm semi-retired and have managed to monetize this content well enough already, I don't work for a big corporation any more..
  15. # I am not a Python programmer
  16. All the code in this repo was initially written by the free version of ChatGPT 4 or Cursor Claude Sonnet3.7 based on short prompts, with no subsequent edits, in a few minutes of my time here and there. I can read Python and mostly make sense of it but I'm not an experienced Python programmer. Look in the relevant issue for a public link to the chat thread that generated the code fro ChatGPT. When I transitioned to Cursor I got the context included as a block comment at the start of each file. This is a ridiculously low friction and easy way to write simple code. Development was migrated to Cursor as it has a much better approach to managing the context of a whole project.
  17. # OKF Bundle Pipeline (New)
  18. This branch adds a complete **Open Knowledge Format (OKF)** pipeline that transforms the author's raw content into a structured, citable knowledge bundle — the "wiki layer" between raw sources and the LLM.
  19. ## What is OKF?
  20. OKF documents are Markdown files with YAML frontmatter. Every document has:
  21. - `type`: content type (Podcast Interview, YouTube Talk, Article, Book, Topic)
  22. - `title`, `description`: human-readable metadata
  23. - `resource`: **the original source URL** (ensures every answer cites its source)
  24. - `tags`: searchable topics
  25. - `timestamp`: publication date
  26. The Concept ID for each document is its file path (e.g., `podcasts/schedulers-with-adrian-cockcroft`).
  27. ## Quick Start
  28. ```bash
  29. git clone https://github.com/adrianco/megpt.git
  30. cd megpt
  31. python3 -m venv venv && source venv/bin/activate
  32. pip install -r requirements.txt
  33. # Set your API key for LLM enrichment (Q&A extraction, topic synthesis)
  34. export ANTHROPIC_API_KEY=sk-ant-...
  35. # Build the full OKF bundle for an author
  36. python build_okf.py virtual_adrianco --author-display "Adrian Cockcroft"
  37. # Build specific content types
  38. python build_okf.py virtual_adrianco podcast --whisper-model tiny
  39. python build_okf.py virtual_adrianco youtube
  40. python build_okf.py virtual_adrianco story
  41. # Process local file archives (Medium/Blogger posts)
  42. python batch_local_files.py authors/virtual_adrianco/medium_adrianco okf/virtual_adrianco/posts
  43. python batch_local_files.py authors/virtual_adrianco/blogger_perfcap_posts okf/virtual_adrianco/posts
  44. # Generate synthesized topic pages (requires ANTHROPIC_API_KEY)
  45. python okf_synthesizer.py virtual_adrianco
  46. # Start the MCP server (for Claude Desktop / Cursor integration)
  47. python mcp_server_v2.py virtual_adrianco
  48. ```
  49. ## OKF Bundle Contents (virtual_adrianco)
  50. The pre-built bundle in `okf/virtual_adrianco/` contains **466 OKF documents**:
  51. | Type | Count | Source |
  52. |------|-------|--------|
  53. | YouTube Talks | 104 | Caption extraction via yt-dlp (no video download) |
  54. | Articles/Posts | 342 | Medium (64), Blogger (264), web articles (14) |
  55. | Podcast Interviews | 20 | Whisper transcription + Simplecast/Libsyn/SoundCloud resolution |
  56. | Books | 2 | PDF extraction (selected page ranges) |
  57. ## Pipeline Architecture
  58. ```
  59. published_content_v2.csv → build_okf.py → processors/
  60. ├── podcast_processor_v2.py (audio → transcript → OKF)
  61. ├── youtube_processor_v2.py (captions → OKF)
  62. ├── story_processor_v2.py (HTML/text → OKF)
  63. └── book_processor_v2.py (PDF → OKF)
  64. ↓
  65. okf/<author>/
  66. ├── podcasts/
  67. ├── talks/
  68. ├── posts/
  69. ├── books/
  70. └── topics/ ← okf_synthesizer.py
  71. ↓
  72. mcp_server_v2.py → Claude Desktop / Cursor
  73. ```
  74. ## Content Manifest (v2 CSV Format)
  75. The enhanced manifest format (`published_content_v2.csv`) adds `SlidesURL` and `Tags` columns:
  76. ```csv
  77. Kind,SubKind,What,Where,Published,URL,SlidesURL,Tags
  78. podcast,episode,Scheduling,Software Engineering Daily,2016,https://...,, scheduling,distributed-systems
  79. youtube,,GOTO 2018 Talk,GOTO Conf,2018,https://...,https://slides.pdf,chaos-engineering
  80. book,"1-50,100-150",My Book,Publisher,2024,https://...,, architecture
  81. ```
  82. ## Citation Requirement
  83. Every OKF document includes a `resource` field with the **original source URL**, and every podcast/talk includes a `## Sources` section. The MCP server includes these citations in all search results, ensuring every answer from the LLM traces back to the original content.
  84. ## MCP Server v2 (OKF-Native)
  85. The new `mcp_server_v2.py` reads OKF documents directly, providing:
  86. - `search(query)`: Keyword search across all 466 docs with source citations
  87. - `get_document(id)`: Full content of any OKF doc
  88. - `list_by_type(type)`: Browse by content type
  89. - `get_topics()`: List synthesized cross-linked topic pages
  90. - `answer_question(question)`: Pre-built prompt with source context
  91. - `summarize_topic(topic)`: Synthesis prompt across the author's body of work
  92. ```json
  93. // Claude Desktop config (~/.claude/claude_desktop_config.json)
  94. {
  95. "mcpServers": {
  96. "megpt-adrianco": {
  97. "command": "python",
  98. "args": ["/path/to/megpt/mcp_server_v2.py", "virtual_adrianco"]
  99. }
  100. }
  101. }
  102. ```
  103. # YouTube Processing
  104. The YouTube processor has been enhanced to handle multiple types of YouTube content automatically:
  105. ## Supported YouTube URL Types
  106. - **Individual Videos**: `https://www.youtube.com/watch?v=VIDEO_ID`
  107. - **Playlists**: `https://www.youtube.com/playlist?list=PLAYLIST_ID`
  108. - **Entire Channels**: `https://www.youtube.com/@username/videos` or `https://www.youtube.com/c/channelname/videos`
  109. ## Processing Features
  110. - **Automatic Detection**: The processor automatically detects whether a URL is an individual video, playlist, or channel
  111. - **Bulk Processing**: Channels and playlists are automatically expanded into individual video entries
  112. - **Consent Page Handling**: Automatically handles YouTube's "Before you continue" consent pages
  113. - **Robust Extraction**: Multiple fallback methods for extracting video metadata and IDs
  114. - **MCP Compatibility**: All videos are saved as individual MCP-compatible JSON files
  115. ## YouTube Downloads
  116. YouTube has strict bot detection measures that can make automatic downloads challenging. The script attempts multiple methods to process YouTube content:
  117. 1. First, it tries to extract video metadata directly from the page HTML
  118. 2. For consent pages, it automatically submits the consent form
  119. 3. Uses multiple regex patterns to find video IDs in the page source
  120. 4. Falls back to alternative extraction methods if standard approaches fail
  121. 5. Creates placeholder entries for channels when individual videos can't be extracted
  122. Despite these measures, YouTube's bot detection is sophisticated and some content may still fail to process automatically. In such cases, the processor will create placeholder entries that reference the original URLs.
  123. For best results, you can try:
  124. - Using a different network (some IP addresses are more likely to trigger bot detection)
  125. - Using a VPN
  126. - Processing from mobile networks (often has fewer restrictions)
  127. - Processing during off-peak hours when YouTube's systems are less restrictive
  128. ## Example Usage
  129. In your `published_content.csv`, you can now include any of these YouTube URL types:
  130. ```csv
  131. Kind,SubKind,What,Where,Published,URL
  132. youtube,,Individual Tech Talk,Conference Name,2024,https://www.youtube.com/watch?v=VIDEO_ID
  133. youtube,,My Tech Talks Playlist,Various Conferences,2024,https://www.youtube.com/playlist?list=PLAYLIST_ID
  134. youtube,,My Complete YouTube Channel,Personal Channel,2024,https://www.youtube.com/@username/videos
  135. ```
  136. The processor will automatically expand playlists and channels into individual video entries, making all content searchable and accessible through the MCP server.
  137. # Enhanced Content Processing
  138. The content processing system has been significantly enhanced to handle complex content structures and archives:
  139. ## File Kind Processing
  140. The system now supports "file" kind entries in CSV that point to directories containing multiple content files:
  141. ```csv
  142. Kind,SubKind,What,Where,Published,URL
  143. file,,Lots of blog post text extracted from medium archive,Medium Archive,2025,./authors/virtual_adrianco/medium_adrianco/
  144. file,,Lots of blog post text extracted from blogger archive,Blogger Archive,2014,./authors/virtual_adrianco/blogger_perfcap_posts/
  145. ```
  146. ### Supported Archive Formats
  147. - **Medium Archives**: Automatically detects `[URL] https://...` format and extracts individual posts
  148. - **Blogger Archives**: Handles `Title: ...\nURL: https://...` format for post extraction
  149. - **Generic Text Files**: Processes any text files with URL extraction capabilities
  150. ### Processing Features
  151. - **Automatic URL Extraction**: Finds URLs in various formats within text files
  152. - **Content Classification**: Assigns appropriate subkinds (medium_post, blogger_post, etc.)
  153. - **Summary Generation**: Creates summaries for substantial content automatically
  154. - **Deduplication**: Prevents processing the same content multiple times
  155. - **Error Handling**: Robust processing with comprehensive logging
  156. ## Content Types Supported
  157. - **YouTube**: Individual videos, playlists, entire channels (135 items)
  158. - **Podcasts**: Episodes with transcripts and summaries (52 items)
  159. - **Books**: PDFs with extracted text and summaries (10 items)
  160. - **Stories**: Narrative content with optional attachments (18 items)
  161. - **Files**: Blog archives, presentations, documents (394 items)
  162. - **InfoQ Videos**: Technical presentations (2 items)
  163. Total: **611 content items** processed and accessible through the MCP server.
  164. # MCP Server Integration
  165. This repository now includes a comprehensive Model Context Protocol (MCP) server that makes the content collection accessible to AI applications. The MCP server transforms the processed content into an AI-accessible knowledge base.
  166. ## Current Content Statistics (virtual_adrianco)
  167. - **Total Items**: 611 content pieces
  168. - **YouTube Videos**: 135 (individual videos, playlists, channels)
  169. - **File Archives**: 394 (Medium posts, Blogger posts, presentations)
  170. - **Podcast Episodes**: 52 (with transcripts and summaries)
  171. - **Stories**: 18 (narrative content)
  172. - **Books**: 10 (PDFs with extracted text and summaries)
  173. - **InfoQ Videos**: 2 (technical presentations)
  174. ## MCP Server Features
  175. - **Content Search**: Semantic search across titles, tags, and sources
  176. - **Content Filtering**: Filter by type, tags, and metadata
  177. - **Analytics**: Collection statistics and content analysis
  178. - **AI Integration**: Compatible with Claude Desktop, Cursor, custom AI agents
  179. - **Multiple Transports**: STDIO, HTTP, and SSE support
  180. ## Usage
  181. ```bash
  182. # Start MCP server (default STDIO mode for Claude Desktop)
  183. python mcp_server.py --author virtual_adrianco
  184. # Start HTTP server for web applications
  185. python mcp_server.py --author virtual_adrianco --transport streamable-http --port 8080
  186. # Generate MCP resources for any author
  187. python create_mcp.py virtual_adrianco
  188. ```
  189. For detailed testing and integration examples, see the `mcp_testing/` directory.
  190. # Building an Author
  191. To use this repo, clone it to a local disk, setup the python environment, run the build.py script for an author and it will walk through the published content table for that author processing each line in turn. The build script will create a downloads/<author> directory and create a state.json file in it which records successful processing steps so that incremental runs of build.py will not re-run the same downloads. Each kind of data needs a corresponding script in the processors directory.
  192. ```
  193. git clone https://github.com/adrianco/megpt.git
  194. cd megpt
  195. python3 -m venv venv
  196. ```
  197. Windows:
  198. ```
  199. venv\Scripts\activate
  200. ```
  201. macOS/Linux:
  202. ```
  203. source venv/bin/activate
  204. ```
  205. ```
  206. pip install -r requirements.txt
  207. ```
  208. If any additional packages are installed during development, regenerate requirements via
  209. ```
  210. pip freeze > requirements.txt
  211. ```
  212. Run the build script
  213. ```
  214. Usage: build.py <author>
  215. python build.py virtual_adrianco
  216. ```
  217. For test purposes process a single kind of data from an arbitrary URL, output to downloads without updating the state.json file
  218. ```
  219. Usage: python process.py <author> <Kind> <SubKind> <URL>
  220. ```
  221. # Creating a RAG based LLM Persona
  222. See this presentation for details on how to get RAGs to work better https://github.com/datastaxdevs/conference-2024-devoxx/
  223. See this issue https://github.com/adrianco/meGPT/issues/11 for my experiments using soopra.ai - currently trained on a subset of this content, and try it out at https://app.soopra.ai/Cockcroft/chat
  224. # Current functional status
  225. Build.py and process.py appear to be operating correctly.
  226. book_processor.py correctly downloads pdfs of books, and extracts page ranges so that relevant sections can be picked out, or mutiple authors can be separated.
  227. Each story download is going to need customized extraction, and the correct div name for The New Stack (thenewstack.io) has been added as a Subkind, and correct text content download is working for the story kind of content.
  228. Medium blog downloads are processed from an archive that can be requested by the author, into text files in the author's medium_posts directory.
  229. Blogger.com archives are processed into text files that include URLs to the original story.
  230. Twitter archives are processed to extract conversations from the archive, ignoring anything other than public tweets that were part of a conversation involving more than one tweet. The archive I used was saved before the naming transition from Twitter to X.
  231. # Notes
  232. I have been assembling my content for a while, and will update the references table and medium download now and again https://github.com/adrianco/meGPT/blob/main/authors/virtual_adrianco/published_content.csv
  233. YouTube videos have transcripts with index offsets into the video itself but the transcript quality isn't good, and they can only be read via API by the owner of the video. It's easier to download videos with pytube and process them with whisper to generate more curated transcripts that identify when the author is talking if there is more than one speaker.
  234. Twitter archive - the raw archive files were over 100MB and too big for github. The extract_conversations script was used to pull out only the tweets that were part of a conversation, so they can be further analyzed to find questions and answers. The code to do this was written by ChatGPT, worked first time, but if there are any problems with the output I'm happy to share the raw tweets. File an issue.
  235. Mastodon archive - available as an RSS feed.
  236. Medium blog platform - available as an RSS feed of the last ten posts. Archive download is processed by code/medium_posts.py to extract text of public posts, ignoring drafts.
  237. Issues have been created to track development of ingestion processing code.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.