Skip to content
Back to search

agent-memory-engine

Memory Engine is a local-first MCP server that gives coding agents persistent, evidence-backed project memory. It automatically builds a project knowledge base, retrieves relevant architecture, constraints, incidents, and source context before complex tasks, and retains verified lessons after successful work.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/uudam42--agent-memory-engine/badge.svg)](/listings/uudam42--agent-memory-engine)

HTML

<a href="/listings/uudam42--agent-memory-engine"><img src="/api/listings/uudam42--agent-memory-engine/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

**macOS / Linux:**

git clone https://github.com/uudam42/agent-memory-engine.git
cd agent-memory-engine
bash scripts/install.sh

**Windows (PowerShell):**

git clone https://github.com/uudam42/agent-memory-engine.git
cd agent-memory-engine
.\scripts\install.ps1

The installer checks Git and Python 3.11+, installs `uv` when needed, resolves dependencies, runs a health check, and prints a ready-to-copy MCP configuration for Cursor or Claude Code. No Docker, WSL, or cloud services required.

Default Deployment Model

Agent Memory Engine runs locally as a **stdio MCP server**. The default setup uses Python, `uv`, and local SQLite/FTS5 storage. Docker, cloud databases, and external embedding services are not required for the standard local workflow.

MCP Stdio Mode *(recommended)*

Runs locally through the MCP client using `uv run`. Supported by Cursor, Claude Code, and any client that implements the MCP stdio transport.

**Option A — explicit project root (Cursor, most clients):**

{
  "mcpServers": {
    "memory-engine": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/agent-memory-engine",
        "memory-engine-mcp",
        "--project-root",
        "/absolute/path/to/your-project"
      ]
    }
  }
}

**Option B — project root via environment variable (Claude Code):**

{
  "mcpServers": {
    "memory-engine": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/agent-memory-engine",
        "memory-engine-mcp"
      ],
      "env": {
        "MEMORY_ENGINE_PROJECT_ROOT": "/absolute/path/to/your-project"
      }
    }
  }
}

> **Note:** Replace all `/absolute/path/to/...` with real paths on your machine. > Config file location and workspace-variable support differ by client: > - **Cursor:** `.cursor/mcp.json` or global Cursor MCP settings > - **Claude Code:** `~/.claude.json` or project-level config > - Run `bash scripts/install.sh` to get a pre-filled config block with your actual paths.

FastAPI / HTTP Mode *(optional)*

A FastAPI application (`memory_engine/main.py`) is included for direct API experimentation, demos, or containerized environments. It is **not required** for the standard MCP workflow.

uvicorn memory_engine.main:app --reload
# API docs at http://localhost:8000/docs

Manual setup *(advanced)*

# Install uv once
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dependencies
uv sync

# Run MCP server directly
uv run memory-engine-mcp --project-root /path/to/project --log-level DEBUG

Open your target project and start coding

That's it. Memory Engine starts automatically on the first MCP connection and handles bootstrap, indexing, and memory management from there.

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 971 lines · 1 finding marked

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <p align="center">
  2. <img src="docs/assets/banner.svg" alt="Agent Memory Engine — local-first MCP runtime for persistent coding agent memory" width="100%" />
  3. </p>
  4. <p align="center">
  5. <strong>Local-first persistent memory and project knowledge runtime for coding agents.</strong>
  6. </p>
  7. <p align="center">
  8. <a href="https://github.com/uudam42/agent-memory-engine/actions/workflows/ci.yml">
  9. </a>
  10. <a href="https://github.com/uudam42/agent-memory-engine/blob/main/LICENSE">
  11. <img src="https://img.shields.io/github/license/uudam42/agent-memory-engine?style=for-the-badge" alt="MIT License" />
  12. </p>
  13. <p align="center">
  14. <a href="README.md">English</a> | <a href="README.zh-CN.md">中文</a>
  15. </p>
  16. ---
  17. # Agent Memory Engine
  18. **A local-first MCP runtime that gives coding agents persistent, evidence-backed project memory and grounded project knowledge across sessions.**
  19. ---
  20. ## Why it exists
  21. Coding agents face a fundamental problem: every session starts cold.
  22. - They forget project context between sessions.
  23. - They repeatedly scan repositories to re-learn what modules do.
  24. - They lose debugging lessons and historical decisions.
  25. - Flat RAG cannot distinguish stable constraints, past incidents, architecture decisions, and raw code evidence.
  26. - Even large context windows still require intelligent prioritization and token budgeting.
  27. Memory Engine solves this by maintaining a structured, evidence-backed memory tree alongside an indexed project knowledge base — both local, both automatic, no infrastructure required.
  28. ---
  29. ## Core capabilities
  30. | Capability | Details |
  31. |---|---|
  32. | **Persistent memory tree** | MemoryNode hierarchy: constraints, architecture, modules, decisions, incidents, procedures |
  33. | **Evidence-backed memory** | Each node links to source Evidence entries (test output, code references, review notes) |
  34. | **Candidate staging** | Reflection generates MemoryCandidates before promoting to the live tree |
  35. | **Confidence-aware promotion** | create / update / merge / supersede / discard / needs_review |
  36. | **Conflict detection** | High-risk areas (auth, schema, state-machine, retry) flagged for review |
  37. | **Ancestor consolidation** | Parent node summaries auto-updated after each promotion |
  38. | **Agent-native recall** | Intent-aware retrieval before coding tasks — no manual queries |
  39. | **Progressive inspection** | Drill down into any memory node, its children, and linked evidence |
  40. | **Automatic post-task reflection** | Agent reports outcome → system decides whether and how to retain knowledge |
  41. | **Knowledge ingestion** | Markdown, code, ADR, test reports, runtime logs, git diffs |
  42. | **Local FTS5 search** | SQLite FTS5 with porter tokenizer; no external search engine |
  43. | **Optional semantic retrieval** | Persistent local sqlite-vec backend with sentence-transformers / Ollama / FastEmbed (ONNX Runtime) embeddings (Phase 13); default OFF, no required deps |
  44. | **Lexical structured fallback** | Full retrieval without vector backend or Docker |
  45. | **Unified ContextPack** | Memory + knowledge merged, deduplicated, token-budgeted |
  46. | **Retrieval traceability** | Per-signal score breakdown in every response |
  47. | **Local-first privacy** | All data stays inside `.memory-engine/`; no telemetry, no cloud calls |
  48. | **Python MCP server** | stdio transport; no TypeScript, no Docker, no external daemon |
  49. | **Zero-touch bootstrap** | Auto-initializes on first MCP connection |
  50. | **Incremental indexing** | JSON manifest; only changed files re-indexed on subsequent runs |
  51. | **Git-aware synchronization** | Detects branch, HEAD commit, staged/modified files via safe read-only Git commands |
  52. | **Branch-aware retrieval** | Prefers memory from the current branch; falls back to mainline then global |
  53. | **Branch-scoped memory writes** | New memories stamped with branch name and scope; mainline promotion is explicit |
  54. | **Multi-granularity memory** | Four retrieval layers (proposition → paragraph → chunk → module summary) created at write-time; selected at query-time by intent |
  55. | **Deterministic proposition extraction** | Atomic facts extracted from docstrings, security comments, raise statements, markdown bullets — no LLM required |
  56. | **Intent-aware granularity routing** | `bug_fix` retrieves constraint/risk propositions; `architecture_review` retrieves module summaries |
  57. | **Query-time context assembly** | Proposition hits optionally expand to parent paragraphs; architecture queries attach module summaries |
  58. | **Memory retention & compaction** | Candidate expiry, stale/superseded archival, multi-source compaction with full lineage — no physical deletion |
  59. | **Protected memory types** | `constraint`, `security_rule`, `architecture`, `decision` excluded from all auto-archive and auto-compaction |
  60. | **Agent memory policy** | Canonical `AGENT_MEMORY_POLICY.md` auto-installed to `CLAUDE.md` / `.cursor/rules/` on first bootstrap; CLI available for manual control |
  61. | **Project context seeding** | `seed_project_context` MCP tool + `memory seed` CLI wizard — write initial constraints, decisions, and project overview on day 1, eliminating the cold-start problem |
  62. | **Windows installer** | PowerShell installer (`scripts/install.ps1`) for Windows-native setup without Docker, WSL, or cloud services |
  63. ---
  64. ## Quick Start
  65. **macOS / Linux:**
  66. ```bash
  67. git clone https://github.com/uudam42/agent-memory-engine.git
  68. cd agent-memory-engine
  69. bash scripts/install.sh
  70. ```
  71. **Windows (PowerShell):**
  72. ```powershell
  73. git clone https://github.com/uudam42/agent-memory-engine.git
  74. cd agent-memory-engine
  75. .\scripts\install.ps1
  76. ```
  77. The installer checks Git and Python 3.11+, installs `uv` when needed, resolves
  78. dependencies, runs a health check, and prints a ready-to-copy MCP configuration
  79. for Cursor or Claude Code. No Docker, WSL, or cloud services required.
  80. ### Default Deployment Model
  81. Agent Memory Engine runs locally as a **stdio MCP server**. The default setup
  82. uses Python, `uv`, and local SQLite/FTS5 storage. Docker, cloud databases, and
  83. external embedding services are not required for the standard local workflow.
  84. ### MCP Stdio Mode *(recommended)*
  85. Runs locally through the MCP client using `uv run`. Supported by Cursor,
  86. Claude Code, and any client that implements the MCP stdio transport.
  87. **Option A — explicit project root (Cursor, most clients):**
  88. ```json
  89. {
  90. "mcpServers": {
  91. "memory-engine": {
  92. "command": "uv",
  93. "args": [
  94. "run",
  95. "--directory",
  96. "/absolute/path/to/agent-memory-engine",
  97. "memory-engine-mcp",
  98. "--project-root",
  99. "/absolute/path/to/your-project"
  100. ]
  101. }
  102. }
  103. }
  104. ```
  105. **Option B — project root via environment variable (Claude Code):**
  106. ```json
  107. {
  108. "mcpServers": {
  109. "memory-engine": {
  110. "command": "uv",
  111. "args": [
  112. "run",
  113. "--directory",
  114. "/absolute/path/to/agent-memory-engine",
  115. "memory-engine-mcp"
  116. ],
  117. "env": {
  118. "MEMORY_ENGINE_PROJECT_ROOT": "/absolute/path/to/your-project"
  119. }
  120. }
  121. }
  122. }
  123. ```
  124. > **Note:** Replace all `/absolute/path/to/...` with real paths on your machine.
  125. > Config file location and workspace-variable support differ by client:
  126. > - **Cursor:** `.cursor/mcp.json` or global Cursor MCP settings
  127. > - **Claude Code:** `~/.claude.json` or project-level config
  128. > - Run `bash scripts/install.sh` to get a pre-filled config block with your actual paths.
  129. ### FastAPI / HTTP Mode *(optional)*
  130. A FastAPI application (`memory_engine/main.py`) is included for direct API
  131. experimentation, demos, or containerized environments. It is **not required**
  132. for the standard MCP workflow.
  133. ```bash
  134. uvicorn memory_engine.main:app --reload
  135. # API docs at http://localhost:8000/docs
  136. ```
  137. ### Manual setup *(advanced)*
  138. ```bash
  139. # Install uv once
  140. curl -LsSf https://astral.sh/uv/install.sh | sh
  141. # Install dependencies
  142. uv sync
  143. # Run MCP server directly
  144. uv run memory-engine-mcp --project-root /path/to/project --log-level DEBUG
  145. ```
  146. ### Open your target project and start coding
  147. That's it. Memory Engine starts automatically on the first MCP connection and
  148. handles bootstrap, indexing, and memory management from there.
  149. ---
  150. ## What happens automatically
  151. ```
  152. User opens project
  153. │
  154. ▼
  155. MCP client starts memory-engine-mcp via stdio
  156. │
  157. ▼
  158. Project root resolved (.git / pyproject.toml / package.json marker)
  159. │
  160. ▼
  161. .memory-engine/ created (if first use)
  162. │
  163. ▼
  164. README, ADRs, architecture docs, constraints indexed first
  165. │
  166. ▼
  167. Broader source files indexed incrementally in background
  168. │
  169. ▼
  170. Agent starts non-trivial coding task
  171. │
  172. ▼ [automatic]
  173. retrieve_agent_context called
  174. │ → relevant constraints, incidents, decisions, procedures, source refs
  175. │
  176. ▼
  177. Agent implements and validates
  178. │
  179. ▼ [automatic, on success]
  180. reflect_and_write called
  181. │ → system evaluates retention gates
  182. │ → creates MemoryCandidates if worthy
  183. │ → promotes to memory tree
  184. │ → consolidates ancestor summaries
  185. │
  186. ▼
  187. memory_status shows updated counts
  188. ```
  189. ---
  190. ## Architecture
  191. ```mermaid
  192. graph TD
  193. A[User / Coding Agent] --> B[MCP Client]
  194. B -->|stdio| C[Python MCP Server]
  195. C --> D[Agent Skills]
  196. C --> E[Service Layer]
  197. C --> F[Knowledge Layer]
  198. D -->|recall / inspect / reflect| E
  199. E -->|memory lifecycle| G[(Memory Tree\nSQLite)]
  200. F -->|ingest / search| H[(Knowledge Base\nSQLite + FTS5)]
  201. F --> I[Optional Vector Backend]
  202. E --> G
  203. F --> H
  204. C --> J[Bootstrap & Incremental Index]
  205. J --> K[.memory-engine/\nproject-local storage]
  206. G --> K
  207. H --> K
  208. ```
  209. ---
  210. ## Agent calling chain
  211. ```mermaid
  212. sequenceDiagram
  213. participant Agent
  214. participant MCP as MCP Server
  215. participant Skills as Agent Skills
  216. participant Services as Service Layer
  217. participant DB as SQLite / FTS5
  218. Agent->>MCP: retrieve_agent_context(task, files, symbols)
  219. MCP->>Skills: QueryAnalyzer.analyze()
  220. Skills->>DB: MemoryNode recall (intent-weighted SQL)
  221. Skills->>DB: KnowledgeSearch (FTS5 + vector RRF)
  222. Skills->>Skills: Rank, compose, dedup, token-trim
  223. MCP-->>Agent: ContextPack (memory + knowledge + trace)
  224. Note over Agent: implements and validates
  225. Agent->>MCP: reflect_and_write(task, outcome, verification_status)
  226. MCP->>Skills: ReflectionSkill.analyze() — gate check
  227. Skills->>Services: PostTaskService.reflect_and_write()
  228. Services->>Services: PromotionService.promote()
  229. Services->>Services: ConsolidationService.update_ancestors()
  230. Services->>DB: persist MemoryNode updates
  231. MCP-->>Agent: {outcome: "persisted", candidates_promoted: 2}
  232. ```
  233. ---
  234. ## Memory lifecycle
  235. ```
  236. Task result
  237. │
  238. ▼
  239. ReflectionSkill.analyze()
  240. │ gates: outcome ≠ failed/reverted, verification_status, confidence ≥ threshold,
  241. │ summary word count, known-trivial patterns
  242. │
  243. ├─ skip → return {skip_reason}
  244. │
  245. └─ pass ▼
  246. │
  247. MemoryCandidate generation
  248. ├─ constraint (importance 0.92)
  249. ├─ procedure (importance 0.72)
  250. ├─ incident/debug (importance 0.85)
  251. ├─ module (importance 0.62)
  252. └─ decision (importance 0.82)
  253. │
  254. ▼
  255. PromotionService.promote()
  256. ├─ create — new node
  257. ├─ update — same title, content refreshed
  258. ├─ merge — near-duplicate (Jaccard ≥ 0.80)
  259. ├─ supersede — existing node confirmed wrong
  260. ├─ discard — low value / already known
  261. └─ needs_review — conflicts with high-confidence existing node
  262. │
  263. ▼
  264. ConsolidationService.update_ancestors()
  265. │ parent.summary = concat(children.summaries)
  266. ▼
  267. cache invalidated + memory_revision bumped
  268. ```
  269. ### Node statuses
  270. | Status | Meaning | Default retrieval |
  271. |---|---|---|
  272. | `candidate` | Staged, pending promotion decision | Excluded |
  273. | `active` | Live, returned in recall | Included |
  274. | `stale` | Outdated; preserved for history | Penalized |
  275. | `superseded` | Replaced by newer node | Excluded |
  276. | `needs_review` | Flagged conflict; human review recommended | Excluded |
  277. | `archived` | Historical / audit-only (Phase 11) | Excluded |
  278. | `compacted` | Synthesis of source memories (Phase 11) | Included with lineage |
  279. | `expired` | Candidate never promoted past window (Phase 11) | Excluded |
  280. ### Phase 11: Memory Retention
  281. Memory does not grow indefinitely. The `MemoryRetentionService` runs lifecycle transitions:
  282. ```bash
  283. # Diagnose
  284. memory retention status <project>
  285. memory retention report <project>
  286. # Apply (dry-run by default)
  287. memory retention run <project>
  288. memory retention run <project> --no-dry-run
  289. # Restore an archived memory
  290. memory retention restore <project> <memory-id>
  291. ```
  292. **Protected types** (`constraint`, `security_rule`, `architecture`, `decision`) are
  293. never auto-archived or auto-compacted.
  294. See [`docs/memory_retention_compaction.md`](docs/memory_retention_compaction.md).
  295. ---
  296. ## Knowledge lifecycle
  297. ```
  298. Documents / code / ADRs / tests / logs / diffs
  299. │
  300. ▼
  301. redact() ← 8 patterns: API keys, tokens, passwords, private keys,
  302. │ connection strings, JWTs, AWS keys, Slack tokens
  303. ▼
  304. SHA-256 content hash → dedup check
  305. │
  306. ▼
  307. Source-type chunker
  308. ├─ Markdown → heading-based sections (≤1200 tokens)
  309. ├─ Code → class/function blocks (≤1000 tokens)
  310. ├─ Test report → result windows
  311. ├─ Log → sliding windows (≤600 tokens)
  312. └─ Diff/Patch → hunk-based chunks
  313. │
  314. ▼
  315. KnowledgeDocument + KnowledgeChunk (SQLite)
  316. │
  317. ├─ FTS5 insert (lexical — always available)
  318. └─ Vector upsert (optional — InMemoryVectorIndex or Qdrant)
  319. │
  320. ▼
  321. hybrid retrieval → RRF fusion → source-quality ranking
  322. │
  323. ▼
  324. UnifiedContextPack (40% of token budget)
  325. ─── Phase 10: multi-granularity write path (runs in parallel) ───────────────
  326. same raw content
  327. │
  328. ├─ paragraph_segmenter → KnowledgeParagraphORM → knowledge_paragraphs_fts
  329. ├─ proposition_extractor → KnowledgePropositionORM → knowledge_propositions_fts
  330. └─ summarizer → KnowledgeChunkSummaryORM → knowledge_summaries_fts
  331. │
  332. ▼
  333. multigranular retrieval (25% of knowledge budget)
  334. │
  335. ▼
  336. UnifiedContextPack.multigranular_chunks (independent of knowledge_chunks budget)
  337. ```
  338. ---
  339. ## Directory structure
  340. ```
  341. memory_engine/
  342. ├── main.py ← FastAPI app (dev / direct API use)
  343. ├── cli.py ← Debug CLI
  344. ├── config.py ← Pydantic Settings
  345. │
  346. ├── agent/ ← Stage 8 namespace (re-exports)
  347. │ ├── skills/ → memory_engine.skills
  348. │ ├── policies/ → reflection gate constants
  349. │ └── contracts/ → agent I/O domain models
  350. │
  351. ├── skills/ ← agent-facing behaviors (recall, inspect, reflect)
  352. ├── services/ ← domain orchestration (promotion, consolidation)
  353. ├── knowledge/ ← ingestion, chunking, FTS5, vector, search, fusion, cache
  354. │ proposition_extractor, paragraph_segmenter, summarizer,
  355. │ granularity_router, multigranular_search (Phase 10)
  356. ├── repositories/ ← persistence abstraction (memory_node, candidate, evidence)
  357. ├── models/ ← Pydantic domain + SQLAlchemy ORM
  358. │
  359. ├── bootstrap/ ← local runtime (project_root, storage, security, state)
  360. ├── runtime/ ← Stage 8 namespace (re-exports bootstrap + cache + config)
  361. │
  362. ├── mcp/ ← MCP adapter (tools, resources, server, project_context)
  363. ├── api/ ← FastAPI routes
  364. └── db/ ← SQLite session + init
  365. docs/
  366. ├── architecture/ ← system-overview, memory-lifecycle, knowledge-pipeline,
  367. │ retrieval-pipeline, mcp-integration, local-runtime,
  368. │ multigranular_memory_architecture (Phase 10)
  369. └── guides/ ← quickstart, configuration, privacy-and-security
  370. tests/
  371. ├── test_phase4.py – test_phase7.py ← phase integration tests
  372. └── test_*.py ← unit and component tests
  373. ```
  374. ---
  375. ## MCP tools
  376. | Tool | Purpose |
  377. |---|---|
  378. | `seed_project_context` | **Call once on new projects.** Write initial constraints, decisions, project description, and conventions directly as active memory nodes. Auto-extracts from README.md when fields are omitted. Eliminates cold-start problem. |
  379. | `retrieve_agent_context` | Retrieve memory + knowledge before a coding task. Phase 10: pass `task_intent`, `preferred_layers`, `proposition_types` to guide granularity routing. |
  380. | `inspect_memory` | Drill into a MemoryNode, its children, and evidence |
  381. | `inspect_knowledge` | Inspect a KnowledgeChunk or source file range (redacted) |
  382. | `reflect_and_write` | Report validated work to the reflection pipeline |
  383. | `memory_status` | Project health, retrieval mode, index counts, revisions |
  384. | `refresh_project_knowledge` | Trigger incremental rescan; also builds Phase 10 proposition/paragraph/summary layers |
  385. ## MCP resources
  386. | Resource | Content |
  387. |---|---|
  388. | `memory://project/current/constraints` | Active project constraints |
  389. | `memory://project/current/architecture` | Architecture and module summaries |
  390. | `memory://project/current/status` | Bootstrap state, retrieval mode, health |
  391. | `memory://project/current/recent-incidents` | Recent debug incidents |
  392. | `memory://project/current/memory-tree-summary` | Memory tree outline |
  393. | `memory://project/current/agent-policy` | Generated agent policy |
  394. | `memory://project/current/git-context` | Current Git state (branch, HEAD, staged/modified files — no remote URLs) |
  395. | `memory://project/current/branch-memory-summary` | Memories organized by branch scope |
  396. | `memory://project/current/sync-status` | Incremental sync and index freshness |
  397. | `memory://project/current/retention-status` | Lifecycle counts, archive/expiry candidates (Phase 11) |
  398. | `memory://project/current/compaction-report` | Compaction group candidates (Phase 11) |
  399. ### Phase 11: Agent Memory Policy
  400. **The policy is installed automatically.** On first bootstrap (first MCP tool
  401. call after setup), Memory Engine writes a workflow policy block directly into
  402. the project's `CLAUDE.md` (Claude Code) and generates
  403. `.memory-engine/generated/AGENT_MEMORY_POLICY.md`. No manual step required.
  404. The policy block contains explicit call/skip decision rules for both tools:
  405. - **`retrieve_agent_context`** — call before editing production code, tests,
  406. CI, dependencies, or config; before debugging; when touching ≥ 2 files or
  407. any unfamiliar subsystem. Skip for pure explanations, single-line typo fixes,
  408. or when already called for the same task this session.
  409. - **`reflect_and_write`** — call after `tests_passed` or `build_success` on
  410. ≥ 2 changed files or a non-trivial architectural decision. Skip if tests
  411. failed, task was reverted, only one trivial file changed, or no validation ran.
  412. For manual control or Cursor adapter installation:
  413. ```bash
  414. # Re-generate policy (idempotent, preserves user content outside markers)
  415. memory policy generate --project-root .
  416. # Install or update client adapters explicitly
  417. memory policy install --project-root . --client claude-code
  418. memory policy install --project-root . --client cursor
  419. # Check adapter installation status
  420. memory policy status --project-root .
  421. ```
  422. See [`docs/agent_memory_policy.md`](docs/agent_memory_policy.md).
  423. ---
  424. ## Local storage
  425. ```
  426. your-project/.memory-engine/
  427. ├── config.yaml ← edit to customize; never overwritten
  428. ├── project_state.json ← bootstrap status, revisions
  429. ├── memory.db ← all data (memories, knowledge, candidates)
  430. ├── indexes/manifests/ ← incremental indexing file manifest
  431. ├── generated/
  432. │ └── AGENT_MEMORY_POLICY.md
  433. ├── bootstrap/bootstrap_report.json
  434. ├── constraints.md ← human-authored; safe to commit
  435. ├── team-rules.md ← human-authored; safe to commit
  436. └── decisions.md ← human-authored; safe to commit
  437. ```
  438. Add `.memory-engine/` to `.gitignore` (generated hint on first bootstrap).
  439. The three human-authored `.md` files may optionally be committed.
  440. **Reset:** `rm -rf your-project/.memory-engine/`
  441. ---
  442. ## Project context seeding (Phase 12)
  443. On a brand-new project the memory database is empty, so `retrieve_agent_context`
  444. returns nothing useful for the first several sessions. Seeding eliminates this
  445. cold-start problem by writing initial memory nodes on day one.
  446. ### Option A — MCP tool `seed_project_context` (agent-driven)
  447. Call **once** when `memory_status` shows `active_memories == 0`:
  448. ```json
  449. {
  450. "description": "Task scheduling system with retry and lifecycle management.",
  451. "constraints": [
  452. "Terminal states (COMPLETED, FAILED) must never be exited.",
  453. "Concurrent execution of the same task is forbidden."
  454. ],
  455. "decisions": [
  456. "SQLite over PostgreSQL — zero-infrastructure local deployment.",
  457. "Event sourcing for audit trail."
  458. ],
  459. "tech_stack": ["Python", "FastAPI", "SQLite"],
  460. "conventions": ["PRs require one reviewer.", "No force-push to main."]
  461. }
  462. ```
  463. All fields are optional. When `description` is omitted, README.md is
  464. auto-scanned for constraint/decision headings and bullet lists (no LLM).
  465. Nodes are written directly as **active** with `confidence=1.0` — authoritative
  466. human input bypasses the candidate/promotion pipeline.
  467. ### Option B — CLI wizard `memory seed` (interactive)
  468. ```bash
  469. memory seed my-project --project-root /path/to/project
  470. ```
  471. Runs a step-by-step prompt for description, tech stack, constraints,
  472. decisions, and conventions, then writes nodes immediately.
  473. ---
  474. ## Phase 13 — Local Persistent Vector Retrieval
  475. Lexical FTS5 retrieval misses cross-wording matches: a query for "DB schema
  476. upgrade" will not rank a chunk that says "database migration". Phase 13 adds a
  477. **persistent local semantic index** so retrieval also matches on meaning, while
  478. keeping the default install lightweight.
  479. ```
  480. Query
  481. ├── FTS5 lexical retrieval (always active)
  482. ├── sqlite-vec semantic retrieval (optional, default OFF)
  483. ├── Branch / revision / lifecycle filters
  484. ├── RRF candidate fusion
  485. ├── Deterministic multi-signal ranker
  486. └── Token-budgeted UnifiedContextPack
  487. ```
  488. **Design guarantees**
  489. - Default `uv sync` installs nothing extra — no model downloads, no new required deps.
  490. - [sqlite-vec](https://github.com/asg017/sqlite-vec) is the persistent backend: no Docker, no external service.
  491. - sentence-transformers, Ollama, and FastEmbed are optional **local** embedding providers. No cloud APIs.
  492. - FTS5 lexical fallback is unchanged when semantic is disabled.
  493. - Branch / revision / lifecycle safety filters are never bypassed by semantic results.
    • Dismissed after review: Tells the agent to set aside its instructions or safety, or to reveal its system prompt: "safety filters are never bypassed"
    • Line 601 on GitHub
  494. - Content is redacted before embedding (same `redact()` path as the rest of the engine).
  495. ### Enable it
  496. **Option A — installer prompt (recommended for new installs)**
  497. The installer asks whether to enable semantic retrieval at the end:
  498. ```bash
  499. bash scripts/install.sh
  500. # → "Enable semantic retrieval? [y/N]"
  501. # → y: installs sentence-transformers + sqlite-vec
  502. ```
  503. Then activate per project (writes to `.memory-engine/config.yaml`, persists across restarts):
  504. ```bash
  505. memory semantic status --enable --project-root /your/project
  506. ```
  507. **Option B — manual**
  508. ```bash
  509. # 1. Install deps (choose one provider)
  510. uv pip install 'memory-engine[semantic-transformers]' # sentence-transformers + sqlite-vec
  511. # or: uv pip install 'memory-engine[semantic-ollama]' # Ollama + sqlite-vec
  512. # or: uv pip install 'memory-engine[semantic-fastembed]' # FastEmbed (ONNX Runtime) + sqlite-vec
  513. # 2. Persist to project config (survives terminal restarts)
  514. memory semantic status --enable --project-root /your/project
  515. ```
  516. **Option C — env vars (CI / temporary override)**
  517. ```bash
  518. export MEMORY_ENGINE_SEMANTIC_ENABLED=1
  519. export MEMORY_ENGINE_EMBEDDING_PROVIDER=sentence_transformers
  520. export MEMORY_ENGINE_EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
  521. ```
  522. Env vars always take precedence over `config.yaml`. When enabled and available,
  523. `retrieval_mode` becomes `hybrid`; otherwise the engine stays in
  524. `lexical_structured_fallback` and `memory_status` returns a `suggestions` field
  525. explaining how to activate it.
  526. ### CLI
  527. ```bash
  528. memory semantic status --project-root . # provider/backend/model + embedded counts
  529. memory semantic doctor --project-root . # availability, dimension, orphan checks
  530. memory semantic reindex --project-root . # incremental embedding (--full to rebuild)
  531. memory semantic clear --project-root . --confirm # clears vectors only
  532. ```
  533. Vectors live in `.memory-engine/vector.db`. `clear` never touches source
  534. knowledge or memory trees. Full details: [docs/local_vector_retrieval.md](docs/local_vector_retrieval.md).
  535. ---
  536. ## Human-authored seed knowledge
  537. Create these files to provide stable project knowledge that cannot be safely
  538. inferred from code alone:
  539. **`.memory-engine/constraints.md`**
  540. ```markdown
  541. # Project Constraints
  542. ## Auth
  543. Do not bypass JWT validation. All routes require Bearer token.
  544. ## Database
  545. Never use raw SQL. SQLAlchemy ORM only.
  546. ## Scheduler
  547. Terminal task states (COMPLETED, FAILED, CANCELLED) are immutable.
  548. ```
  549. **`.memory-engine/team-rules.md`**
  550. ```markdown
  551. # Team Rules
  552. - PRs require 2 approvals before merge
  553. - All public APIs must have OpenAPI documentation
  554. - Log structured JSON only (no print statements in production code)
  555. ```
  556. These files are indexed as high-priority knowledge on bootstrap and returned in
  557. context before relevant tasks.
  558. ---
  559. ## Retrieval modes
  560. ### Default local mode: `lexical_structured_fallback`
  561. Active when no persistent vector backend is available (default for local use).
  562. Signals used for ranking:
  563. - SQLite FTS5 lexical match (BM25)
  564. - Module-path overlap with current task files
  565. - Symbol overlap with current task symbols
  566. - Memory tree proximity
  567. - Node importance and confidence
  568. - Freshness (recency weighting)
  569. - Project-scoped TTL cache
  570. ### Enhanced mode: `hybrid` (Phase 13)
  571. Active when semantic retrieval is enabled and both sqlite-vec and a local
  572. embedding provider are available. Adds real cosine similarity over chunk,
  573. paragraph, proposition, and summary embeddings via RRF fusion. The
  574. `semantic_similarity` score in the retrieval trace reflects the actual vector
  575. score (no longer always 0.0). See the Phase 13 section above.
  576. **Vector retrieval is optional.** The default local mode works without
  577. sqlite-vec, Qdrant, Docker, or any external service.
  578. ### Phase 10: multi-granularity retrieval
  579. Active automatically when Phase 10 knowledge is indexed (created by `refresh_project_knowledge` or the normal ingest path).
  580. Four retrieval layers, created deterministically at write-time (no LLM):
  581. | Layer | Unit | Typical use |
  582. |---|---|---|
  583. | Proposition | One atomic fact (`shell=False is enforced`) | Bug fixes, security audits |
  584. | Paragraph | One function or heading section | Feature work, code explanation |
  585. | Chunk | Fixed-size content block (existing) | General keyword search |
  586. | Module summary | Whole-file digest with key symbols | Architecture review, onboarding |
  587. The layer selected for a query is determined by `task_intent`:
  588. ```python
  589. # in retrieve_agent_context:
  590. task_intent = "bug_fix" # → propositions first (constraint/security_rule/risk)
  591. task_intent = "architecture_review" # → module summaries first
  592. task_intent = "feature_implementation" # → paragraphs + propositions + summaries
  593. ```
  594. Callers can also override directly:
  595. ```python
  596. preferred_layers = ["proposition"] # force proposition-only
  597. proposition_types = ["security_rule"] # sub-filter by type
  598. ```
  599. Results appear in `multigranular_chunks` alongside the existing `knowledge_chunks`.
  600. See [`docs/architecture/multigranular_memory_architecture.md`](docs/architecture/multigranular_memory_architecture.md) for the full design.
  601. ---
  602. ## Privacy and security
  603. - **Local-only:** all data stays in `.memory-engine/`; nothing leaves your machine
  604. - **No telemetry:** no usage data sent anywhere
  605. - **No cloud embedding:** no external API calls by default
  606. - **No Docker:** not required for any feature
  607. - **Path boundaries:** all file reads restricted to resolved project root
  608. - **Symlink protection:** links escaping project root rejected
  609. - **Secret redaction:** runs before persistence and before MCP output
  610. - **Default exclusions:** `.env`, `secrets/`, `*.pem`, `*.key`, `node_modules/`, `.git/`, binary files, files over 5 MB
  611. - **No auto Git commits:** never — Git is read-only (`rev-parse`, `branch`, `status`, `merge-base` only)
  612. - **No remote URL exposure:** Git remote URLs are never returned in any output
  613. - **No Git identity exposure:** user name and email are never collected or returned
  614. - **No destructive Git commands:** `commit`, `reset`, `push`, `clean`, `checkout`, `merge`, `rebase`, `fetch` are all blocked
  615. - **No writes outside `.memory-engine/`:** guaranteed
  616. ---
  617. ## Configuration
  618. Generated at `.memory-engine/config.yaml` on first bootstrap:
  619. ```yaml
  620. project:
  621. name: auto
  622. root_path: auto
  623. runtime:
  624. auto_bootstrap: true
  625. auto_recall: true
  626. auto_reflect: true
  627. auto_index_on_start: true
  628. incremental_indexing: true
  629. privacy:
  630. mode: local
  631. redact_secrets: true
  632. allow_network_embedding: false
  633. knowledge:
  634. include:
  635. - README.md
  636. - docs/**
  637. - src/**
  638. - app/**
  639. - lib/**
  640. - tests/**
  641. exclude:
  642. - node_modules/**
  643. - .git/**
  644. - .venv/**
  645. - dist/**
  646. - build/**
  647. - .env
  648. - secrets/**
  649. max_file_size_mb: 5
  650. retrieval:
  651. default_token_budget: 6000
  652. cache_enabled: true
  653. vector_backend: auto
  654. allow_degraded_fallback: true
  655. # Phase 9: branch-aware retrieval
  656. branch_aware_ranking: true
  657. prefer_current_branch: true
  658. include_ancestor_branch_memory: true
  659. include_mainline_fallback: true
  660. runtime:
  661. # Phase 9: Git-aware synchronization
  662. git_aware_sync: true
  663. check_git_status_on_retrieval: true
  664. auto_incremental_sync: true
  665. memory:
  666. # Phase 9: branch-scoped write policy
  667. branch_scope_on_feature_work: current_branch
  668. mainline_promotion_requires_confirmation: true
  669. privacy:
  670. # Phase 9: Git identity protection
  671. expose_git_remote_url: false
  672. redact_git_identity: true
  673. ```
  674. User edits are preserved on re-bootstrap.
  675. ---
  676. ## Demo scenario
  677. **Scheduler project. Task:** Add exponential retry backoff without breaking terminal task state semantics.
  678. 1. Agent calls `retrieve_agent_context`:
  679. ```json
  680. {
  681. "constraints": [
  682. {
  683. "title": "Terminal State Immutability",
  684. "summary": "COMPLETED, FAILED, CANCELLED are terminal states. Any operation that transitions out of a terminal state is a critical bug.",
  685. "importance": 0.95
  686. }
  687. ],
  688. "incidents": [
  689. {
  690. "title": "Retry Loop Re-entered Terminal Task",
  691. "summary": "In v0.8.2, a retry race condition re-entered a COMPLETED task. Root cause: retry check did not verify terminal status before re-queuing.",
  692. "importance": 0.88
  693. }
  694. ],
  695. "knowledge_chunks": [
  696. {
  697. "source_path": "docs/adr/003-retry-policy.md",
  698. "preview": "Decision: use exponential backoff with jitter. Max 5 retries..."
  699. }
  700. ],
  701. "retrieval_trace": [...],
  702. "meta": {
  703. "retrieval_mode": "lexical_structured_fallback",
  704. "vector_backend": "ephemeral",
  705. "warnings": ["Semantic vector retrieval is unavailable..."]
  706. }
  707. }
  708. ```
  709. 2. Agent implements retry logic with terminal-state guard.
  710. 3. Tests pass. Agent calls `reflect_and_write`:
  711. ```json
  712. {
  713. "outcome": "persisted",
  714. "candidates_promoted": 2,
  715. "consolidation_notes": ["Parent 'Scheduler Core' summary updated"]
  716. }
  717. ```
  718. ---
  719. ## Debug CLI
  720. For maintainers, demos, and troubleshooting only. Not the normal workflow.
  721. ```bash
  722. memory-engine debug status --project-root /path/to/project
  723. memory-engine debug bootstrap --project-root /path/to/project
  724. memory-engine debug index --project-root /path/to/project
  725. memory-engine debug recall "add retry backoff" --project-root /path/to/project
  726. memory-engine debug inspect <node-id>
  727. memory-engine debug reset-project --project-root /path/to/project
  728. ```
  729. ---
  730. ## Testing
  731. ```bash
  732. # Run all tests
  733. pytest -v
  734. # Run focused
  735. pytest tests/test_phase7.py -v
  736. pytest tests/test_phase6.py -v
  737. pytest -k "recall" -v
  738. ```
  739. 511 tests currently passing. All deterministic. No external services required.
  740. (Semantic retrieval tests use mocked providers or `pytest.importorskip("sqlite_vec")`
  741. — no models are downloaded during the test run.)
  742. ---
  743. ## Limitations and future work
  744. - **Persistent local vector backend** — current InMemoryVectorIndex does not survive process restarts
  745. - **Optional Qdrant backend** — interface exists; client not installed by default; enables semantic cross-lingual retrieval
  746. - **PyPI publishing** — `pip install memory-engine-mcp` not yet available
  747. - **Binary packaging** — no binary installer yet
  748. - **Streamable HTTP remote mode** — stdio only; no team-shared HTTP transport yet
  749. - **Team-shared memory** — each project has isolated local storage; no shared team memory yet
  750. - **Authentication and permissions** — no per-user or per-team access control yet
  751. - **Richer code parsing** — chunking is line-range based; AST-aware parsing is future work
  752. - **Larger repository benchmarks** — not yet validated at monorepo scale
  753. - **Retention scheduling** — `memory retention run` is explicit; no background daemon
  754. ---
  755. ## Contributing
  756. 1. Read [`docs/architecture/system-overview.md`](docs/architecture/system-overview.md) first.
  757. 2. Preserve service-layer boundaries: MCP/API layers stay thin.
  758. 3. Keep business logic in `skills/`, `services/`, `knowledge/`.
  759. 4. New knowledge source types → `knowledge/chunkers.py` + new `SourceType` enum value.
  760. 5. New MCP tools → `mcp/tools.py`, thin wrapper only; delegate to services.
  761. 6. New domain model → `models/domain.py` or `models/knowledge_domain.py`.
  762. 7. Add tests for all new behavior.
  763. 8. Never weaken project-root security boundaries.
  764. 9. Run `pytest -v` and confirm all tests pass before opening a PR.
  765. ---
  766. ## Development
  767. ```bash
  768. # Install dev dependencies (includes pytest, httpx)
  769. uv sync --extra dev
  770. # Run tests
  771. uv run pytest -v
  772. # Start FastAPI service (dev / direct API use — not required for MCP)
  773. uvicorn memory_engine.main:app --reload
  774. # API docs at http://localhost:8000/docs
  775. # Run MCP server directly (stdio, blocks until client disconnects)
  776. uv run memory-engine-mcp --project-root /path/to/project --log-level DEBUG
  777. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.