Skip to content
Back to search

mfs

A context harness for AI agents: all your scattered context — code, memory, docs, databases, SaaS — in one searchable, browsable, file-like interface.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/zilliztech--mfs/badge.svg)](/listings/zilliztech--mfs)

HTML

<a href="/listings/zilliztech--mfs"><img src="/api/listings/zilliztech--mfs/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Start the server

**Server** — start it in one terminal:

- **Option A — uv tool** (no clone; installs from PyPI, reads local paths directly)

uv tool install mfs-server && mfs-server run

Wait until the server logs show `Application startup complete` before switching to the client terminal.

- **Option B — Docker Compose** (isolated; builds the image from source)

git clone https://github.com/zilliztech/mfs.git && cd mfs

docker compose -f deployments/compose/docker-compose.yml up --build

The first build takes a few minutes. The ~600 MB embedding model downloads on the first `mfs add` and is cached in the `mfs-data` volume.

Install the CLI

**Client** — in another terminal, install the CLI and point it at the server:

cargo install mfs-cli

Set environment variables

# server address
export MFS_API_URL=http://127.0.0.1:13619

# option A: reads ~/.mfs/server.token automatically — no export needed
# option B: read the token the container auto-generated
export MFS_API_TOKEN="$(docker compose -f deployments/compose/docker-compose.yml exec -T mfs-server cat /data/server.token)"

Create a tiny sample tree:

mkdir -p /tmp/hello-mfs
echo "Print a greeting to the user." > /tmp/hello-mfs/greet.txt

Then index and search it:

mfs add --upload /tmp/hello-mfs
# --upload bundles + sends the tree to the container
# (omit it under option A: the server reads the path in place)
mfs search "where the greeting is printed" /tmp/hello-mfs

> macOS: run `xattr -d com.apple.quarantine $(which mfs)` once if prompted about an unidentified developer.

Skill install

Install the skills once:

# every project + every supported agent (drop -g for the current project only)
npx skills add zilliztech/mfs --all -g

<details> <summary>Install to a specific agent</summary>

Pass one or more `-a <agent>`:

npx skills add zilliztech/mfs -a claude-code -a codex -g

`<agent>` can be `claude-code`, `codex`, `cursor`, `windsurf`, `github-copilot`, `gemini-cli`, `opencode`, `zed`, `cline`, `continue` — 70+ agents in all.

</details>

<details> <summary>Check for updates</summary>

npx skills check
npx skills update

For project-level installs, re-run the `npx skills add` command to update.

</details>

📥 Then open your agent (Claude Code, Codex, …) and ask in plain language. First, ingest something:

Use the mfs-ingest skill to spin up a tiny hello-world project in /tmp/hello-mfs
and ingest it

🔍 Then search and read across it:

Use the mfs-find skill to find where the greeting is printed in the hello-mfs
project, and show me the exact lines

🎉 That's it — you're up and running. From here, point MFS at your real sources.

> 🛠️ **The first run is a one-time setup, and the agent walks you through it:** > it installs the `mfs` CLI, helps you start a local server, and downloads a > ~600 MB local embedding model into `~/.mfs/`. Give it a minute. After that the > whole stack runs locally and offline — **no API key, no GPU, no cloud account.** > > Don't want the model download? If you already have an embedding-service API key > (OpenAI, …), point the server at it instead — see > [Configure the server](#-configure-the-server).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 839 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <h1 align="center"><img src="https://github.com/user-attachments/assets/1a14c3e3-32c3-4474-a081-ce737bfc439a" alt="MFS logo" width="48" align="absmiddle" /> MFS — Multi-source File-like Search</h1>
  2. <p align="center">
  3. <strong>A context harness for AI agents — and for building them: one unified workspace over your code, memory, skills, docs, messages, and every data source.</strong>
  4. </p>
  5. <p align="center">
  6. <a href="https://github.com/zilliztech/mfs/blob/main/LICENSE"><img src="https://img.shields.io/github/license/zilliztech/mfs?style=flat-square" alt="License"></a>
  7. <a href="https://crates.io/crates/mfs-cli"><img src="https://img.shields.io/crates/v/mfs-cli?style=flat-square&color=orange&logo=rust&logoColor=white" alt="crates.io"></a>
  8. <img src="https://img.shields.io/badge/python-%3E%3D3.10-blue?style=flat-square&logo=python&logoColor=white" alt="Python">
  9. <a href="https://github.com/zilliztech/mfs/actions/workflows/lint.yml"><img src="https://img.shields.io/github/actions/workflow/status/zilliztech/mfs/lint.yml?branch=main&style=flat-square&label=lint" alt="Lint"></a>
  10. <a href="https://milvus.io/"><img src="https://img.shields.io/badge/powered%20by-Milvus-00A1EA?style=flat-square" alt="Milvus"></a>
  11. <a href="https://zilliztech.github.io/mfs/"><img src="https://img.shields.io/badge/docs-mfs-blue?style=flat-square&logo=readthedocs&logoColor=white" alt="Docs"></a>
  12. <a href="https://discord.com/invite/FG6hMJStWu"><img src="https://img.shields.io/badge/Discord-chat-7289da?style=flat-square&logo=discord&logoColor=white" alt="Discord"></a>
  13. <a href="https://x.com/zilliz_universe"><img src="https://img.shields.io/badge/follow-%40zilliz__universe-000000?style=flat-square&logo=x&logoColor=white" alt="X"></a>
  14. <a href="https://github.com/zilliztech/mfs/stargazers"><img src="https://img.shields.io/github/stars/zilliztech/mfs?style=flat-square" alt="Stars"></a>
  15. </p>
  16. ---
  17. <p align="center">
  18. <img src="https://github.com/user-attachments/assets/42c4e26c-c26a-463f-bd97-c5bb2d38eabe" alt="MFS multi-source analysis demo" width="880" />
  19. </p>
  20. Modern AI agents run on a huge amount of **context** — and it's sprawling,
  21. scattered, and siloed across dozens of sources and formats. That makes it hard
  22. to manage and harder to locate: for any given task, which slice of all that
  23. context is the part that actually matters? It piles up in:
  24. - **Code & repos** — every repo the agent reads or writes
  25. - **Memory & skills** — past-session transcripts, memory `.md` files, reusable `SKILL.md` skills
  26. - **Docs & knowledge** — PDFs, design specs, Notion, crawled web pages
  27. - **Chat & mail** — Slack, Discord, Gmail, Feishu
  28. - **Issues & CRM** — Jira, Linear, HubSpot, Zendesk
  29. - **Databases & object stores** — Postgres, Mongo, BigQuery, Snowflake, S3, Drive
  30. MFS gathers it under one shell: every source becomes a **file-like tree under a
  31. stable URI**, driven by two skills your agent loads:
  32. - **🗂️ mfs-ingest** — the skill that bundles the commands for bringing sources
  33. in: `mfs add` registers and indexes any connector (re-run to re-sync), and
  34. `mfs connector` lists, inspects, and removes them.
  35. - **🔎 mfs-find** — the skill that bundles the commands for finding across
  36. what's ingested, in two families:
  37. - **Search** — `search` and `grep` find things fast across huge volumes.
  38. - **Browse** — `ls`, `cat`, `tree`, `head`, `tail` give progressive, precise
  39. navigation down to the exact bytes.
  40. <p align="center">
  41. <img src="https://github.com/user-attachments/assets/1430d872-4184-4fb3-9168-a0b715dc621a" alt="MFS architecture: clients (CLI, SDKs, agent skills) talk to mfs-server, which unifies many context sources into one searchable namespace" width="880" />
  42. </p>
  43. ## 🚀 Quick start
  44. ### Start the server
  45. **Server** — start it in one terminal:
  46. - **Option A — uv tool** (no clone; installs from PyPI, reads local paths directly)
  47. ```bash
  48. uv tool install mfs-server && mfs-server run
  49. ```
  50. Wait until the server logs show `Application startup complete` before switching to the client terminal.
  51. - **Option B — Docker Compose** (isolated; builds the image from source)
  52. ```bash
  53. git clone https://github.com/zilliztech/mfs.git && cd mfs
  54. docker compose -f deployments/compose/docker-compose.yml up --build
  55. ```
  56. The first build takes a few minutes. The ~600 MB embedding model downloads on the first `mfs add` and is cached in the `mfs-data` volume.
  57. ### Install the CLI
  58. **Client** — in another terminal, install the CLI and point it at the server:
  59. ```bash
  60. cargo install mfs-cli
  61. ```
  62. ### Set environment variables
  63. ```bash
  64. # server address
  65. export MFS_API_URL=http://127.0.0.1:13619
  66. # option A: reads ~/.mfs/server.token automatically — no export needed
  67. # option B: read the token the container auto-generated
  68. export MFS_API_TOKEN="$(docker compose -f deployments/compose/docker-compose.yml exec -T mfs-server cat /data/server.token)"
  69. ```
  70. Create a tiny sample tree:
  71. ```bash
  72. mkdir -p /tmp/hello-mfs
  73. echo "Print a greeting to the user." > /tmp/hello-mfs/greet.txt
  74. ```
  75. Then index and search it:
  76. ```bash
  77. mfs add --upload /tmp/hello-mfs
  78. # --upload bundles + sends the tree to the container
  79. # (omit it under option A: the server reads the path in place)
  80. mfs search "where the greeting is printed" /tmp/hello-mfs
  81. ```
  82. > macOS: run `xattr -d com.apple.quarantine $(which mfs)` once if prompted about an unidentified developer.
  83. ### Skill install
  84. Install the skills once:
  85. ```bash
  86. # every project + every supported agent (drop -g for the current project only)
  87. npx skills add zilliztech/mfs --all -g
  88. ```
  89. <details>
  90. <summary>Install to a specific agent</summary>
  91. Pass one or more `-a <agent>`:
  92. ```bash
  93. npx skills add zilliztech/mfs -a claude-code -a codex -g
  94. ```
  95. `<agent>` can be `claude-code`, `codex`, `cursor`, `windsurf`, `github-copilot`,
  96. `gemini-cli`, `opencode`, `zed`, `cline`, `continue` — 70+ agents in all.
  97. </details>
  98. <details>
  99. <summary>Check for updates</summary>
  100. ```bash
  101. npx skills check
  102. npx skills update
  103. ```
  104. For project-level installs, re-run the `npx skills add` command to update.
  105. </details>
  106. 📥 Then open your agent (Claude Code, Codex, …) and ask in plain language. First,
  107. ingest something:
  108. ```text
  109. Use the mfs-ingest skill to spin up a tiny hello-world project in /tmp/hello-mfs
  110. and ingest it
  111. ```
  112. 🔍 Then search and read across it:
  113. ```text
  114. Use the mfs-find skill to find where the greeting is printed in the hello-mfs
  115. project, and show me the exact lines
  116. ```
  117. 🎉 That's it — you're up and running. From here, point MFS at your real sources.
  118. > 🛠️ **The first run is a one-time setup, and the agent walks you through it:**
  119. > it installs the `mfs` CLI, helps you start a local server, and downloads a
  120. > ~600 MB local embedding model into `~/.mfs/`. Give it a minute. After that the
  121. > whole stack runs locally and offline — **no API key, no GPU, no cloud account.**
  122. >
  123. > Don't want the model download? If you already have an embedding-service API key
  124. > (OpenAI, …), point the server at it instead — see
  125. > [Configure the server](#-configure-the-server).
  126. ## 💡 Use cases
  127. ### 🧠 Your agent's memory and skills
  128. Past-session memory files (`.md`, `.jsonl`) and reusable
  129. skills become one searchable namespace — the prompt you tuned last week or a
  130. decision logged three sessions ago, one query away.
  131. > The `mfs` line is the CLI; the trailing `#` comment is the same request to an
  132. > agent (`/mfs-ingest` or `/mfs-find` — Codex uses `$mfs-ingest` / `$mfs-find`
  133. > instead of `/`).
  134. ```bash
  135. mfs add path/to/memory # /mfs-ingest index my session memory
  136. mfs add path/to/skills # /mfs-ingest index my skills
  137. mfs search "the prompt I tuned for refund disputes" --all # /mfs-find the refund-dispute prompt
  138. ```
  139. <details>
  140. <summary>Output</summary>
  141. ```text
  142. file://local/.agents/memory/2026-05-31.jsonl score=0.88
  143. {"role":"note","text":"refund-dispute prompt: lead with the order ID, then ..."}
  144. file://local/.agents/skills/support-triage/SKILL.md score=0.74
  145. ## Refund disputes — confirm the order ID first, then check the gateway log ...
  146. ```
  147. Open the hit — `mfs cat file://local/.agents/memory/2026-05-31.jsonl`:
  148. ```text
  149. {"role":"note","text":"refund-dispute prompt: lead with the order ID, then ask
  150. for the gateway transaction id; never promise a timeline before the dispute
  151. window is confirmed."}
  152. ```
  153. </details>
  154. ### 💻 Your codebases
  155. Index the repos your agent reads or writes and grep them by meaning — find the
  156. helper by what it *does*, not the name you've forgotten.
  157. ```bash
  158. mfs add path/to/repo # /mfs-ingest index my repos
  159. mfs search "where do we retry failed webhook deliveries?" path/to/repo # /mfs-find our webhook retry logic
  160. ```
  161. <details>
  162. <summary>Output</summary>
  163. ```text
  164. file://local/repos/payments/webhooks/deliver.go score=0.84
  165. 87 // cap exponential backoff at 6 attempts, then dead-letter
  166. 88 func (d *Dispatcher) retryDelivery(ev Event) error {
  167. file://local/repos/payments/webhooks/deliver_test.go score=0.69
  168. TestRetryDelivery_DeadLettersAfterMaxAttempts ...
  169. ```
  170. Open the matching lines — `mfs cat file://local/repos/payments/webhooks/deliver.go --range 80:110`:
  171. ```text
  172. // retryDelivery re-sends a webhook with capped exponential backoff.
  173. func (d *Dispatcher) retryDelivery(ev Event) error {
  174. for attempt := 1; attempt <= maxAttempts; attempt++ {
  175. if err := d.post(ev); err == nil {
  176. return nil
  177. }
  178. time.Sleep(backoff(attempt))
  179. }
  180. return d.deadLetter(ev) // give up after maxAttempts
  181. }
  182. ```
  183. </details>
  184. ### 📄 Documents, images, any format
  185. PDFs, Word docs, Markdown, screenshots — MFS converts each to text **locally**
  186. (PDF / docx → Markdown, no API key), and with a vision model on it describes
  187. images too. One search spans every format.
  188. ```bash
  189. mfs add path/to/docs # /mfs-ingest index my design docs
  190. mfs add path/to/screenshots # /mfs-ingest index my screenshots
  191. mfs search "audit-log retention and the dashboards that show it" --all # /mfs-find audit-log retention + dashboards
  192. ```
  193. <details>
  194. <summary>Output</summary>
  195. ```text
  196. file://local/design-docs/data-governance.pdf score=0.86
  197. ... Audit logs are retained for 400 days, then moved to cold storage; access
  198. beyond 90 days requires a break-glass approval ...
  199. file://local/screenshots/grafana-2026-06-02.png score=0.71
  200. A Grafana dashboard; the p99 latency panel climbs to ~800 ms around 14:10,
  201. well above the 200 ms band on the other panels ...
  202. ```
  203. Open the hit — `mfs cat file://local/design-docs/data-governance.pdf`:
  204. ```text
  205. # Data Governance — Audit Logs
  206. Audit logs are retained for **400 days**, then moved to cold storage. Access
  207. beyond 90 days requires a break-glass approval recorded in the change log.
  208. ```
  209. </details>
  210. ### ☁️ Cloud drives and buckets
  211. Mount a Google Drive or S3 bucket; its files become searchable text alongside
  212. your local ones — no syncing, no downloads.
  213. ```bash
  214. mfs add gdrive://my-drive --config ./gdrive.toml # /mfs-ingest add my google drive
  215. mfs add s3://acme-exports --config ./s3.toml # /mfs-ingest add our s3 exports bucket
  216. mfs search "the Q3 board deck" --all # /mfs-find the Q3 board deck
  217. ```
  218. <details>
  219. <summary>Output</summary>
  220. ```text
  221. gdrive://my-drive/Board/2026-Q3-review.pdf score=0.87
  222. ... Q3 highlights: net revenue retention 118%, two enterprise logos closed ...
  223. s3://acme-exports/finance/2026-q3-summary.csv score=0.70
  224. quarter,net_revenue,nrr,churn 2026Q3,4.2M,1.18,1.4% ...
  225. ```
  226. Open the hit — `mfs cat gdrive://my-drive/Board/2026-Q3-review.pdf`:
  227. ```text
  228. # 2026 Q3 Board Review
  229. - Net revenue retention: 118%
  230. - New enterprise logos: Globex, Initech
  231. - Gross margin: 79% (+2 pts QoQ)
  232. ```
  233. </details>
  234. ### 🌍 Online sources
  235. Crawl a docs site or mount a GitHub repo with its issues — remote content lands
  236. in the same namespace as your local files.
  237. ```bash
  238. mfs add web://docs.your-product.com # /mfs-ingest crawl our docs site
  239. mfs add github://your-org/your-repo --config ./github.toml # /mfs-ingest add our github repo
  240. mfs search "how do we rotate signing keys?" --all # /mfs-find signing-key rotation
  241. ```
  242. <details>
  243. <summary>Output</summary>
  244. ```text
  245. web://docs.your-product.com/security/key-rotation score=0.88
  246. ... Signing keys rotate every 90 days. Trigger an early rotation from the
  247. admin console; the previous key stays valid for a 24-hour overlap ...
  248. github://your-org/your-repo/issues.jsonl score=0.75
  249. #312 "Automate signing-key rotation" state=open labels=[security]
  250. ```
  251. Open the hit — `mfs cat web://docs.your-product.com/security/key-rotation`:
  252. ```text
  253. # Rotating signing keys
  254. Signing keys rotate automatically every 90 days. To rotate early, open
  255. Admin → Security → Keys and click "Rotate now"; the previous key stays valid
  256. for a 24-hour overlap so in-flight tokens keep verifying.
  257. ```
  258. </details>
  259. ### 💬 Team chat and tickets
  260. Mount Slack, Gmail, Jira, Linear and pull the thread, the ticket, and the email
  261. behind a decision into one answer.
  262. ```bash
  263. mfs add slack://acme --config ./slack.toml # /mfs-ingest add our slack
  264. mfs add jira://acme --config ./jira.toml # /mfs-ingest add our jira
  265. mfs search "why did we revert the burst guard?" --all # /mfs-find why we reverted the burst guard
  266. ```
  267. <details>
  268. <summary>Output</summary>
  269. ```text
  270. slack://acme/channels/platform/messages.jsonl score=0.90
  271. [Tue 09:40] @carol: reverting the burst guard — it dropped healthy traffic
  272. [Tue 09:42] @dave: agreed, reopening PLAT-491 to re-tune the window
  273. jira://acme/teams/PLAT/issues.jsonl score=0.81
  274. PLAT-491 "rate-limit guard misfires under burst" state=Reopened
  275. ```
  276. Read the ticket — `mfs cat jira://acme/teams/PLAT/issues.jsonl --locator '{"id":"PLAT-491"}'`:
  277. ```text
  278. PLAT-491 rate-limit guard misfires under burst state=Reopened
  279. assignee: dave priority: high
  280. The burst guard dropped healthy traffic during the Tuesday spike; reverted in
  281. PR #604. Re-tuning the window before re-enabling.
  282. ```
  283. </details>
  284. ### 🎫 Customers and support
  285. Pull your CRM and help desk together — the account, its open tickets, and the
  286. call notes behind a customer issue in one query.
  287. ```bash
  288. mfs add hubspot://acme --config ./hubspot.toml # /mfs-ingest add our hubspot crm
  289. mfs add zendesk://acme --config ./zendesk.toml # /mfs-ingest add our zendesk
  290. mfs search "why is Globex unhappy with onboarding?" --all # /mfs-find Globex onboarding issues
  291. ```
  292. <details>
  293. <summary>Output</summary>
  294. ```text
  295. zendesk://acme/tickets.jsonl score=0.88
  296. #5821 "Onboarding blocked on SSO setup" status=open priority=high
  297. requester ops@globex.com — "third week without working SSO ..."
  298. hubspot://acme/companies/globex/notes.jsonl score=0.74
  299. call note: Globex renewal at risk; onboarding friction flagged by the CSM ...
  300. ```
  301. Read the ticket — `mfs cat zendesk://acme/tickets.jsonl --locator '{"id":5821}'`:
  302. ```text
  303. #5821 Onboarding blocked on SSO setup status=open priority=high
  304. requester: ops@globex.com
  305. "Third week without working SSO — the SAML metadata upload keeps failing with a
  306. 500. This is blocking our rollout."
  307. ```
  308. </details>
  309. ### 🗄️ Production data
  310. Point MFS at Postgres, Mongo, or BigQuery and search rows as text — each row is a
  311. file-like object, so `mfs cat` pulls back the full record.
  312. ```bash
  313. mfs add postgres://prod/orders --config ./pg.toml # /mfs-ingest add the prod orders table
  314. mfs search "refunds stuck in pending over 7 days" postgres://prod/orders # /mfs-find stuck pending refunds
  315. ```
  316. <details>
  317. <summary>Output</summary>
  318. ```text
  319. postgres://prod/orders score=0.79
  320. {"id":"ord_8842","status":"pending","refund_requested_at":"2026-05-30",
  321. "amount":129.00,"gateway":"stripe","note":"customer disputed, awaiting ..."}
  322. ```
  323. Read the full row — `mfs cat postgres://prod/orders --locator '{"id":"ord_8842"}'`:
  324. ```text
  325. id ord_8842
  326. status pending
  327. refund_requested_at 2026-05-30
  328. amount 129.00
  329. gateway stripe
  330. note customer disputed, awaiting gateway confirmation
  331. ```
  332. </details>
  333. ### 🌐 One query across all of them at once
  334. With a few sources registered, `--all` fans one query across all of them — files,
  335. databases, trackers, chat — in a single result shape, so any hit copies straight
  336. into `mfs cat`.
  337. ```bash
  338. mfs search "rate-limit guard misfires under burst" --all # /mfs-find the burst rate-limit bug
  339. ```
  340. <details>
  341. <summary>Output</summary>
  342. ```text
  343. slack://acme/channels/oncall/messages.jsonl score=0.91
  344. [Mon 22:14] @alice: ratelimiter pegged 500ms p99 tail, dump attached
  345. [Mon 22:18] @bob: smells like the burst guard from PR #418
  346. jira://acme/teams/PLAT/issues.jsonl score=0.83
  347. PLAT-491 "rate-limit guard misfires under burst" state=In Progress
  348. file://local/repo/src/throttle.go score=0.71
  349. 42 func handleRateLimit(req Request) error {
  350. ```
  351. Copy any hit into a read — same locators everywhere:
  352. ```bash
  353. mfs cat ./repo/src/throttle.go --range 42:78
  354. mfs cat jira://acme/teams/PLAT/issues.jsonl --locator '{"id":"PLAT-491"}'
  355. ```
  356. </details>
  357. First time adding a source and unsure what its TOML needs? Don't write it by hand
  358. — just tell your agent which source you want to index, and the **mfs-ingest**
  359. skill walks you through which credentials to grab (and where), then writes the
  360. config and registers it for you.
  361. ## 🧰 Commands
  362. | Group | Command | What it does |
  363. |---|---|---|
  364. | **Ingest & manage** | **`mfs add <path\|uri> [--config x.toml]`** | register a source and index it; re-run to re-sync |
  365. | | `mfs connector probe <uri> --config x.toml` | dry-run a connection before registering |
  366. | | `mfs connector list` · `inspect` · `remove` | see and manage what's registered |
  367. | **Search** | **`mfs search "<query>" [<path\|uri>] [--all]`** | hybrid semantic + keyword; scope to a path/URI, or `--all` |
  368. | | `mfs grep <pattern> <path>` | exact keyword / full-text |
  369. | **Browse & read** | `mfs ls <uri>` · `mfs tree <uri>` | list one level, or a whole subtree |
  370. | | **`mfs cat <uri> [--range a:b] [--locator '{...}']`** | read the exact bytes, or one structured record |
  371. | | `mfs head <uri>` · `mfs tail <uri>` | sample the first / last entries |
  372. | **Operate** | `mfs status` | server, connectors, and jobs at a glance |
  373. | | `mfs job list` · `show <id>` · `cancel <id>` | track indexing jobs |
  374. | | `mfs config show` | resolved endpoint / profile / token |
  375. | | `mfs serve start` · `stop` | manage a local server process |
  376. **Browse and search are complementary — no fixed order:**
  377. - **Browse** (`ls` · `cat` · `tree`) — no index, fast and exact; best for walking
  378. a small or local tree.
  379. - **Search** (`search` · `grep`) — needs an upfront index, then finds things fast
  380. across huge volumes with fuzzy matching; best for rough filtering.
  381. Through an agent it's the same set in natural language, split across two skills:
  382. - **`mfs-find`** — search + browse: `search` · `grep` · `ls` · `cat` · `tree` ·
  383. `head` · `tail`.
  384. - **`mfs-ingest`** — register + manage: `add` · `connector probe / list / inspect
  385. / remove`.
  386. ## 🔌 Connectors
  387. Local files are only the start. MFS speaks to a growing catalog of sources —
  388. object stores, databases, code hosts, issue trackers, CRMs, chat, mail, docs —
  389. and mounts each one as a **URI tree** you `ls` / `cat` / `grep` / `search` like a
  390. local directory. Same verbs, same result shape, everywhere.
  391. | Category | Source | URI prefix | What you search |
  392. |---|---|---|---|
  393. | 📁 Files & objects | Local files | `file://` | any folder — text, Markdown, code, PDF, docx, images |
  394. | | Amazon S3 (& R2 / GCS / MinIO) | `s3://` | bucket objects, converted to text |
  395. | | Google Drive | `gdrive://` | Docs, Sheets, PDFs and files in a Drive |
  396. | 🗄️ Databases | PostgreSQL | `postgres://` | tables and rows as searchable records |
  397. | | MySQL | `mysql://` | tables and rows as searchable records |
  398. | | MongoDB | `mongo://` | collections and documents |
  399. | | BigQuery | `bigquery://` | datasets and tables |
  400. | | Snowflake | `snowflake://` | databases and tables |
  401. | 💻 Code & issues | GitHub | `github://` | repo files, issues, and PRs |
  402. | | Jira | `jira://` | projects, issues, comments |
  403. | | Linear | `linear://` | teams, issues, comments |
  404. | 🧑‍💼 CRM & support | HubSpot | `hubspot://` | contacts, companies, deals, notes |
  405. | | Zendesk | `zendesk://` | support tickets and comments |
  406. | 💬 Chat & mail | Slack | `slack://` | channels and message history |
  407. | | Discord | `discord://` | servers, channels, threads |
  408. | | Gmail | `gmail://` | mail threads and messages |
  409. | | Feishu / Lark | `feishu://` | docs and messages |
  410. | 🌐 Docs & web | Notion | `notion://` | pages and databases |
  411. | | Web | `web://` | crawled pages, converted to Markdown |
  412. New connectors slot in behind the same interface, so the catalog keeps growing.
  413. Each needs a small TOML (credentials + what to expose) — three ways to get it
  414. right:
  415. - **Ask your agent** — describe the source in plain language; the `mfs-ingest`
  416. skill finds the credentials, says where to get them, and writes the TOML.
  417. - **Probe first** — `mfs connector probe <uri> --config x.toml` dry-runs a
  418. config, no registration and no writes.
  419. - **Read the reference** — each connector documents its exact fields and auth.
  420. > The connector TOML never holds a raw secret — it carries a **reference**
  421. > (`env:VAR` or `file:/path`), and the actual token, password, or credential
  422. > file lives on the **server** (its process env or a mounted file). So the CLI
  423. > and your agent never touch raw credentials, and a hosted MFS can have each user
  424. > supply their own through the service instead of a file on disk.
  425. ## 🏗️ Architecture
  426. Everything above runs the simplest way — client and server on one machine. The
  427. same MFS also scales to production; here's how it's put together.
  428. MFS is a **thin client over a stateful server**, talking over one HTTP `/v1` API:
  429. - **Client** — the `mfs` CLI, the SDKs, and the agent skills (`mfs-find` /
  430. `mfs-ingest`). Stateless, so re-creating it on a laptop, a CI runner, or an
  431. agent runtime is free.
  432. - **Server** — the setup wizard, all config / credentials / env vars, the queue
  433. + workers, and the data backends. Everything that matters lives here, so
  434. `env:` / `file:` secret references always resolve on the server, never the
  435. client.
  436. ```text
  437. ┌────────────────┐ ┌────────────────────────────────────────┐
  438. │ CLIENT │ │ SERVER · mfs-server │
  439. │ ────────────── │ │ ────────────────────────────────────── │
  440. │ mfs CLI │ │ setup wizard │
  441. │ SDKs │ │ queue + workers │
  442. │ skills │ │ config · env vars · credentials │
  443. │ · mfs-find │ ── HTTP /v1 ──▶ │ │
  444. │ · mfs-ingest │ │ backends (scale up as needed): │
  445. └────────────────┘ │ vector Milvus Lite → Zilliz Cloud │
  446. │ metadata SQLite → Postgres │
  447. │ caches local filesystem │
  448. └────────────────────────────────────────┘
  449. ```
  450. The only real deployment choice is **where the server runs**. Move it onto its
  451. own host (a VM or a single container), or scale it out across a Docker Compose
  452. stack or a Kubernetes cluster — the CLI and skills stay with you either way.
  453. The table below is a **recommended layout** per mode; for how to actually set
  454. each piece, see [Configure the server](#-configure-the-server).
  455. | Piece | Local (one machine) | Single host (its own VM or container) | Distributed (Compose / Kubernetes) |
  456. |---|---|---|---|
  457. | `mfs` CLI | your machine | your machine | your machine |
  458. | Agent skills | your machine | your machine | your machine |
  459. | `mfs-server` + workers | your machine | the server host | the server cluster (api + worker pods) |
  460. | `mfs-server setup` wizard | your machine | the server host | the server cluster |
  461. | `server.toml` | your machine | the server host | the server cluster (ConfigMap / mounted file) |
  462. | Connector credentials + secret files | your machine | the server host | the server cluster (Docker / k8s secrets) |
  463. | `env:` / `file:` ref values | your machine | the server host | the server cluster (pod env / mounted files) |
  464. | Vector DB | Milvus Lite (local file) | self-hosted Milvus or Zilliz Cloud | Zilliz Cloud |
  465. | Metadata DB | SQLite (local file) | Postgres | Postgres |
  466. | `file://` ingest | server reads the path in place | CLI bundles + uploads the tree | CLI bundles + uploads the tree |
  467. > That last row is automatic: on a shared filesystem the server reads local paths
  468. > directly; otherwise the CLI bundles and uploads them — no flags needed.
  469. Through an agent you don't think about any of this — the skills already encode
  470. it. The agent detects whether client and server share a machine and adjusts
  471. (local read vs upload) on its own, so there's nothing about deployment mode to
  472. spell out in a prompt. Just use MFS normally.
  473. For a split deployment, point the CLI at the server and you're set:
  474. ```bash
  475. export MFS_API_URL=https://mfs.your-corp.internal
  476. export MFS_API_TOKEN=...
  477. mfs status
  478. ```
  479. Docker images, a Compose file, and a Helm chart for split api / worker
  480. deployments live under [`deployments/`](deployments/).
  481. **🏭 Built for production.** MFS was built for production from day one — not a weekend demo. The split design
  482. scales to **production-scale** data: pair the vector backend with
  483. [Zilliz Cloud](https://cloud.zilliz.com/signup?utm_source=github&utm_medium=referral&utm_campaign=mfs-readme)'s Vector Lakebase and MFS indexes and searches
  484. massive corpora, with the reliability and [design choices below](#-features).
  485. ## 🔧 Configure the server
  486. Almost all configuration lives **on the server** — the embedding provider, the
  487. vector backend, the metadata database, caches, auth, and every connector's
  488. credentials all sit in the server's `server.toml`. The **client** barely has any
  489. config: a tiny TOML that just says *which server to talk to* (endpoint + token),
  490. so pointing the CLI at a local or a remote server is the whole client story.
  491. There are two ways to configure the server, and they write the same `server.toml`:
  492. - **The wizard** — `mfs-server setup` is a convenient interactive walk through
  493. the common choices (embedding provider, vector DB, database, cache, auth).
  494. Defaults are self-contained, so you can press Enter through to a working local
  495. server and opt into hosted backends (OpenAI, Zilliz Cloud, Postgres, …) only
  496. where you need them.
  497. - **The TOML** — everything the wizard writes, plus the advanced knobs it
  498. doesn't surface (cache size / eviction, chunker thresholds, namespace, worker
  499. counts, per-connector tuning), lives in `~/.mfs/server.toml`. Edit it directly
  500. for anything beyond the basics.
  501. > Both run **on the server's machine or container**, not on the client — they
  502. > write the server's `server.toml`. In local mode (client and server on the same
  503. > machine), that's simply your local machine.
  504. <p align="center">
  505. <img src="https://github.com/user-attachments/assets/2adc8090-76e2-4073-aae8-776ca4ba541e" alt="mfs-server setup wizard demo" width="820" />
  506. </p>
  507. ```bash
  508. mfs-server setup # full interactive walkthrough
  509. mfs-server setup --section embedding # re-run just one section
  510. ```
  511. What each section of `server.toml` configures:
  512. | Section | Configures | Default | Swap in |
  513. |---|---|---|---|
  514. | `[embedding]` | embedding provider · model · dim | local ONNX (BGE-M3 int8, no key) | `openai` · `gemini` · `voyage` · `jina` · `mistral` · `ollama` · `local` — needs that provider's key/extra |
  515. | `[milvus]` | vector store | Milvus Lite (file under `~/.mfs/`) | self-hosted Milvus or [Zilliz Cloud](https://cloud.zilliz.com/signup?utm_source=github&utm_medium=referral&utm_campaign=mfs-readme) — set `uri` + `token` |
  516. | `[database]` | metadata + transformation-cache DB | SQLite (file) | Postgres — set `dsn` |
  517. | `[artifact_cache]` | converted-blob cache | local fs under `~/.mfs/cache` | size / eviction knobs |
  518. | `[description]` | image descriptions (vision LLM) | off | `openai` · `anthropic` · `gemini` — needs key |
  519. | `[summary]` | directory / file summaries | off | provider + model, `dir` / `file` scope |
  520. | `auth_token` | API auth | auto-generated `server.token` | a fixed token, or `-` to disable |
  521. Secrets never sit in the TOML as plaintext — the TOML carries a **reference**
  522. (`env:VAR` or `file:/path`) and the real value lives on the **server** (its
  523. process env or a mounted credential file), resolved at runtime. The same holds
  524. for every connector's credentials, so secrets stay on the server, never on the
  525. client.
  526. <details>
  527. <summary>⭐ Get a free Zilliz Cloud cluster (URI + token)</summary>
  528. For a fully managed vector backend,
  529. [sign up](https://cloud.zilliz.com/signup?utm_source=github&utm_medium=referral&utm_campaign=mfs-readme)
  530. for a free Zilliz Cloud cluster — the console gives you the **public endpoint
  531. (URI)** and an **API key (token)**. Put them in `[milvus]` as `uri` + `token`,
  532. keeping the key out of the TOML with an `env:` ref.
  533. ![Sign up and get the URI and API key](https://raw.githubusercontent.com/zilliztech/claude-context/master/assets/signup_and_get_apikey.png)
  534. </details>
  535. <details>
  536. <summary>Use OpenAI instead of the local model?</summary>
  537. Set the embedding provider to OpenAI and export your key — no model download:
  538. ```toml
  539. [embedding]
  540. provider = "openai" # instead of the default local "onnx"
  541. ```
  542. ```bash
  543. export OPENAI_API_KEY=sk-...
  544. ```
  545. </details>
  546. ## ✨ Features
  547. - **🗂️ One file-like interface, any source.** Whatever the source or format, it
  548. becomes a single file-like tree under a stable URI. Agents already speak shell,
  549. so there's no new query language and no per-source SDK — the same handful of
  550. verbs reach everything, and what you learn once carries across every connector.
  551. - **🌐 Your whole workspace, one interface.** Stop wiring a separate context
  552. harness for every scenario. Memory, code, docs, chat, tickets, databases — MFS
  553. unifies your entire working context, with its history and state, into one
  554. search-and-browse interface.
  555. - **🔍 Search and browse, two legs of one loop.** Hybrid semantic + keyword
  556. search locates fast across huge volumes; progressive browse then narrows to the
  557. exact bytes or rows. Together they lift precise recall *and* cut token spend —
  558. you pull in only what matters, and never trust a hit until you've reopened it.
  559. - **🛡️ Local and production, equally at home.** Run fully local and offline, or
  560. at production scale — neither is an afterthought. Every component is swappable
  561. and independently scalable, so the same MFS moves between the two by
  562. configuration alone. The index is derived and crash-safe: upstream stays the
  563. source of truth, so you can delete it and rebuild from the original sources,
  564. losing nothing.
  565. - **🤖 Agent-native.** Built for how agents actually work — especially context
  566. and memory management — so it slots into any agent setup. And when you're
  567. building an agent of your own, you can build it on top of MFS too.
  568. ## 🤖 Build agent applications on top of MFS
  569. Beyond using MFS through an agent in your daily work, you can build **on** it —
  570. treat MFS as the retrieval/context **base layer** your own agent application sits
  571. on. The robust index pipeline and the broad connector catalog are already handled:
  572. you never touch embeddings, the vector store, or per-source plumbing — you point
  573. at MFS and focus on the app on top.
  574. ```text
  575. ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
  576. │ agents │ │ your app code │ │ your skills/MCP/ │
  577. │ → skills + CLI │ │ → SDK (Py / TS) │ │ plugins → CLI │
  578. │ (use directly) │ │ (BUILD ON IT) │ │ (BUILD ON IT) │
  579. └────────┬─────────┘ └────────┬─────────┘ └────────┬─────────┘
  580. │ │ │
  581. └──────────────────────┼──────────────────────┘
  582. ▼
  583. ┌────────────────────────────────────────────────────────────────┐
  584. │ MFS — CLI · SDK · HTTP /v1 │
  585. └───────────────────────────────┬────────────────────────────────┘
  586. │
  587. ▼
  588. ┌────────────────────────────────────────────────────────────────┐
  589. │ index pipeline · chunking · embedding · vector DB · caches │
  590. │ a catalog of source connectors (MFS owns all of this) │
  591. └────────────────────────────────────────────────────────────────┘
  592. ```
  593. - **SDK** — call the generated Python / TypeScript clients under
  594. [`sdks/`](sdks/) straight from your application code (long-running daemons,
  595. services, language runtimes without a shell).
  596. - **CLI** — call the `mfs` command to build your own **skills**, **MCP servers**,
  597. or **plugins** on top of the same surface.
  598. ### 🧩 Examples: well-known projects, rebuilt on MFS
  599. Every example under [`examples/`](examples/) replicates a well-known project on
  600. top of MFS — the same capability, in a fraction of the code — because MFS
  601. already owns the part that used to be the hard part for each of them:
  602. ingestion, chunking, embedding, the vector store, and hybrid search across many
  603. source types. What's left is thin glue, or a single skill describing a strategy.
  604. - [`claude-context`](examples/claude-context/) — a replica of
  605. [claude-context](https://github.com/zilliztech/claude-context) ("make the
  606. codebase the context for any coding agent"): an MCP server exposing
  607. `search` + `read` over MFS, about 60 lines.
  608. - [`deep-research-skill`](examples/deep-research-skill/) — the "reason and
  609. search over private data" job [deep-searcher](https://github.com/zilliztech/deep-searcher)
  610. does as a standalone framework, done here as a single skill on top of
  611. `mfs-find` — no vector-DB glue, no provider matrix, no custom
  612. retrieval-loop code.
  613. - [`open-tag-skill`](examples/open-tag-skill/) — a self-hosted homage to
  614. [Claude Tag](https://www.anthropic.com/news/introducing-claude-tag),
  615. Anthropic's hosted `@Claude` Slack teammate: `@mention` a bot in Slack —
  616. `@OpenClaude` or `@OpenCodex` — and it gathers authorized MFS context and
  617. runs the task on a CLI agent backend.
  618. ![Open Tag — delegating a PR review across Slack channels](https://github.com/user-attachments/assets/6cb1db05-dd12-4a13-a9fa-1a1bf69bcf28)
  619. See [`examples/README.md`](examples/README.md) for the full list, or each
  620. example's own README for its install command and walkthrough.
  621. ## 🗺️ Roadmap
  622. - **Multiple processing profiles** — run different sources through different
  623. pipelines from one server: a code-tuned embedding model for your repos, a
  624. multilingual one for your docs. Because models differ in dimension and vector
  625. space, each profile maps to its own collection (a single one can't mix them),
  626. and you search within a profile or across all.
  627. - **Multi-user credentials and access control** — per-user secrets and
  628. permissions, so a shared or hosted MFS can serve many users safely.
  629. - **A wider connector catalog** — the set of sources keeps growing.
  630. ## 🙏 Acknowledgements
  631. MFS is shaped by several related projects:
  632. - [claude-context](https://github.com/zilliztech/claude-context),
  633. [memsearch](https://github.com/zilliztech/memsearch), and
  634. [deep-searcher](https://github.com/zilliztech/deep-searcher) — earlier
  635. Zilliz code-search, memory-search, and deep-research efforts whose
  636. community feedback shaped MFS's agent-facing direction.
  637. - [VKFS](https://github.com/ZeroZ-lab/vkfs) — a sister exploration
  638. of a Unix-like interface for agent access to vector-backed
  639. knowledge.
  640. ## ⚖️ License
  641. Apache-2.0. See [LICENSE](LICENSE).
examples/open-tag-skill/open-tag/scripts/slack_socket_agent.py · Bundled script · 176 lines · 2 findings marked
  1. #!/usr/bin/env python3
  2. from __future__ import annotations
  3. import argparse
  4. import os
  5. import re
  6. import subprocess
  7. import tempfile
  8. from pathlib import Path
  9. from typing import Any
  10. from slack_bolt import App
  11. from slack_bolt.adapter.socket_mode import SocketModeHandler
  12. MENTION_RE = re.compile(r"<@[^>]+>")
  13. def require_env(name: str) -> str:
  14. value = os.getenv(name)
  15. if not value:
  16. raise RuntimeError(f"{name} is required")
  17. return value
  18. def skill_dir() -> Path:
  19. return Path(__file__).resolve().parents[1]
  20. def default_workdir() -> Path:
  21. return Path(os.getenv("OPENTAG_WORKDIR", str(Path.cwd())))
  22. def strip_mention(text: str) -> str:
  23. stripped = MENTION_RE.sub("", text).strip()
  24. return stripped or "Find the most relevant context for this thread and summarize it."
  25. def build_thread_text(client: Any, channel: str, thread_ts: str) -> str:
  26. response = client.conversations_replies(channel=channel, ts=thread_ts, limit=30)
  27. lines = []
  28. for message in response.get("messages", []):
  29. user = message.get("user") or message.get("bot_id") or "unknown"
  30. text = message.get("text", "")
  31. lines.append(f"{user}: {text}")
  32. return "\n".join(lines)
  33. def run_backend(backend: str, channel: str, question: str, thread_text: str, timeout: int) -> str:
  34. with tempfile.NamedTemporaryFile("w", suffix=".txt", encoding="utf-8", delete=False) as f:
  35. f.write(thread_text)
  36. thread_file = Path(f.name)
  37. cmd = [
  38. "python3",
  39. str(skill_dir() / "scripts" / "opentag_agent.py"),
  40. "--backend",
  41. backend,
  42. "--channel-id",
  43. channel,
  44. "--question",
  45. question,
  46. "--thread-file",
  47. str(thread_file),
  48. "--skill-dir",
  49. str(skill_dir()),
  50. "--workdir",
  51. str(default_workdir()),
  52. "--timeout",
  53. str(timeout),
  54. ]
  55. try:
  56. result = subprocess.run(
  57. cmd,
  58. check=False,
  59. text=True,
  60. stdout=subprocess.PIPE,
  61. stderr=subprocess.STDOUT,
  62. timeout=timeout + 10,
  63. )
  64. output = result.stdout.strip()
  65. if result.returncode != 0:
  66. return f"Open Tag backend failed with exit code {result.returncode}:\n```text\n{output[-3000:]}\n```"
  67. if len(output) > 12000:
  68. output = output[-12000:]
  69. return output or "Open Tag finished without output."
  70. finally:
  71. try:
  72. thread_file.unlink()
  73. except OSError:
  74. pass
  75. def suggested_bot_name(backend: str) -> str:
  76. if os.getenv("OPENTAG_BOT_NAME"):
  77. return os.environ["OPENTAG_BOT_NAME"]
  78. return {"claude": "OpenClaude", "codex": "OpenCodex"}.get(backend, "OpenTag")
  79. def print_live_summary(backend: str) -> None:
  80. bot = suggested_bot_name(backend)
  81. scopes = [s.strip() for s in os.getenv("MFS_ALLOWED_SCOPES", "").split(",") if s.strip()]
  82. channel = os.getenv("SLACK_CHANNEL_ID", "(not set)")
  83. invoke = {
  84. "claude": "claude -p --dangerously-skip-permissions",
  85. "codex": "codex exec --dangerously-bypass-approvals-and-sandbox",
  86. }[backend]
  87. print("=" * 64)
  88. print(f"Open Tag is live as @{bot}")
  89. print(f" Brain : {backend} -> {invoke}")
  90. print(f" Memory : {len(scopes)} permitted MFS scope(s):")
  91. for scope in scopes or ["(none — set MFS_ALLOWED_SCOPES)"]:
  92. print(f" - {scope}")
  93. print(f" Slack : listening for @mentions in channel {channel}")
  94. print("")
  95. print(" Anyone who can @mention the bot in that channel can drive the")
  96. print(" backend, which runs with your shell and inherited environment.")
  97. print(" Slack text flows into the prompt, so treat every mention as")
  98. print(" untrusted input: use an isolated channel on a non-production host.")
  99. print(f" Invite the bot only where it should respond: /invite @{bot}")
  100. print("=" * 64)
  101. def create_app(backend: str, timeout: int) -> App:
  102. app = App(token=require_env("SLACK_BOT_TOKEN"))
  103. @app.event("app_mention")
  104. def handle_mention(event: dict[str, Any], client: Any, logger: Any) -> None:
  105. channel = event["channel"]
  106. thread_ts = event.get("thread_ts") or event["ts"]
  107. question = strip_mention(event.get("text", ""))
  108. status = client.chat_postMessage(
  109. channel=channel,
  110. thread_ts=thread_ts,
  111. text=f"Open Tag is working on this with `{backend}`.",
  112. )
  113. try:
  114. thread_text = build_thread_text(client, channel, thread_ts)
  115. answer = run_backend(backend, channel, question, thread_text, timeout)
  116. client.chat_update(
  117. channel=channel,
  118. ts=status["ts"],
  119. text=answer,
  120. )
  121. except Exception as exc: # noqa: BLE001
  122. logger.exception("Open Tag failed")
  123. answer = f"Open Tag failed: `{type(exc).__name__}: {exc}`"
  124. client.chat_update(channel=channel, ts=status["ts"], text=answer)
  125. return app
  126. def main() -> None:
  127. parser = argparse.ArgumentParser(description="Run Open Tag Slack Socket Mode bridge.")
  128. parser.add_argument(
  129. "--backend",
  130. choices=["claude", "codex"],
  131. default=os.getenv("OPENTAG_BACKEND"),
  132. )
  133. parser.add_argument(
  134. "--timeout", type=int, default=int(os.getenv("OPENTAG_TIMEOUT_SECONDS", "420"))
  135. )
  136. args = parser.parse_args()
  137. if not args.backend:
  138. parser.error("--backend or OPENTAG_BACKEND is required")
  139. app = create_app(args.backend, args.timeout)
  140. print_live_summary(args.backend)
  141. SocketModeHandler(app, require_env("SLACK_APP_TOKEN")).start()
  142. if __name__ == "__main__":
  143. main()

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.