Skip to content
Back to search

LLM-Agents-Ecosystem-Handbook

One-stop handbook for building, deploying, and understanding LLM agents with 60+ skeletons, tutorials, ecosystem guides, and evaluation tools.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/oxbshw--llm-agents-ecosystem-handbook/badge.svg)](/listings/oxbshw--llm-agents-ecosystem-handbook)

HTML

<a href="/listings/oxbshw--llm-agents-ecosystem-handbook"><img src="/api/listings/oxbshw--llm-agents-ecosystem-handbook/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 344 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <div align="center">
  2. # LLM Agents Ecosystem Handbook
  3. **A practical operating manual for building, evaluating, securing, and shipping modern LLM agent systems.**
  4. [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)
  5. [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
  6. [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md)
  7. [![LLM-Friendly](https://img.shields.io/badge/llms.txt-ready-blue.svg)](llms.txt)
  8. [![Providers](https://img.shields.io/badge/providers-24%2B-blueviolet)](providers/README.md)
  9. </div>
  10. ---
  11. > Modern agents are not "a prompt + a tool." They are **systems** — with identity, memory, skills, tools, MCP integrations, guardrails, observability, evals, and a provider strategy. This handbook teaches the whole stack and ships templates, blueprints, runnable adapters, and curated examples you can adopt today.
  12. ## What's in this repo
  13. A curated, opinionated, **production-oriented** handbook in seven parts:
  14. 1. **Concepts** — Agent OS, identity, memory, skills, MCP, safety, observability — every layer of the modern agent stack
  15. 2. **Provider ecosystem** — adapters + docs for **24+ LLM providers** (frontier APIs, fast inference, marketplaces, enterprise clouds, specialty, local runtimes), with a router for fallback chains
  16. 3. **Skills ecosystem** — design guide, taxonomy, maturity model, security checklist, and a curated skill catalog
  17. 4. **Prompt engineering** — agent prompt patterns, instruction hierarchy, context engineering, prompt-injection defense
  18. 5. **Coding-agent workflows** — for Claude Code, Cursor, Codex, Aider, Cline, and custom runtimes — repo instructions, prompts, review checklist, safe refactoring
  19. 6. **Design docs** — agent / technical design docs, ADR guide, design reviews, rollout plans, the `DESIGN.md` machine-readable spec
  20. 7. **Curated catalog** — 100+ existing agent skeletons, framework comparisons, evaluation tools, tutorials — preserved and improved
  21. ## Who this is for
  22. | You are… | Start at |
  23. |---|---|
  24. | New to agents | [docs/beginners_guide.md](docs/beginners_guide.md) → [agent_os/README.md](agent_os/README.md) |
  25. | Building a production agent | [blueprints/](blueprints/) → [checklists/production_readiness_checklist.md](checklists/production_readiness_checklist.md) |
  26. | Picking / wiring providers | [providers/README.md](providers/README.md) → [providers/provider_matrix.md](providers/provider_matrix.md) |
  27. | Comparing frameworks | [docs/framework_comparison.md](docs/framework_comparison.md) |
  28. | Adding memory / RAG | [memory/](memory/) → [tutorials/rag_tutorials](tutorials/rag_tutorials) |
  29. | Adding MCP | [mcp/](mcp/) → [mcp/mcp_security.md](mcp/mcp_security.md) |
  30. | Designing Skills | [skills/](skills/) → [skills/skill_design_guide.md](skills/skill_design_guide.md) |
  31. | Working with coding agents | [coding_agents/](coding_agents/) → [coding_agents/prompts/](coding_agents/prompts/) |
  32. | Writing better prompts | [prompt_engineering/](prompt_engineering/) |
  33. | Designing & rolling out | [design_docs/](design_docs/) |
  34. | Hardening safety/evals | [safety/](safety/) → [evals/](evals/) |
  35. | Coding agent reading this repo | [llms.txt](llms.txt) → [llm_wiki/index.md](llm_wiki/index.md) |
  36. ---
  37. ## Modern Agent Stack
  38. | Layer | Purpose | Where in this repo |
  39. |---|---|---|
  40. | **Model / Provider** | LLM choice + abstraction + routing | [providers/](providers/) |
  41. | **Orchestration** | Agent loops, planning, handoffs | [docs/framework_comparison.md](docs/framework_comparison.md), [blueprints/](blueprints/) |
  42. | **Tool** | Function calling and external actions | [agent_os/mcp_layer.md](agent_os/mcp_layer.md) |
  43. | **MCP** | Standardized external context and tools | [mcp/](mcp/) |
  44. | **Memory** | Durable user/project/semantic memory | [memory/](memory/) |
  45. | **Skills** | Reusable, progressive-loading workflows | [skills/](skills/) |
  46. | **Identity** | Personality, mission, refusal style | [agent_os/agent_identity.md](agent_os/agent_identity.md), [templates/](templates/) |
  47. | **Prompt** | System prompt design, instruction hierarchy, defenses | [prompt_engineering/](prompt_engineering/) |
  48. | **Safety** | Guardrails, approvals, policy | [safety/](safety/) |
  49. | **Observability** | Tracing, spans, cost, latency, evals | [observability/](observability/), [evals/](evals/) |
  50. | **Deployment** | Shipping agents to production | [design_docs/rollout_plan.md](design_docs/rollout_plan.md) |
  51. | **Coding-agent harness** | Claude Code, Cursor, Codex, Aider, Cline | [coding_agents/](coding_agents/) |
  52. 📖 Deep dive: [agent_os/README.md](agent_os/README.md)
  53. ---
  54. ## Provider ecosystem
  55. The handbook ships an `LLMProvider` abstraction with **24+ providers** across six families. Most providers go through a single OpenAI-compatible code path; specialty / local providers are first-class.
  56. | Provider type | Examples | Best for |
  57. |---|---|---|
  58. | **Frontier APIs** | OpenAI, Anthropic, Google Gemini | Reasoning, tool use, production agents |
  59. | **Fast inference** | Groq, Cerebras, SambaNova | Low-latency workloads |
  60. | **Marketplaces** | OpenRouter, Together, Fireworks, DeepInfra | Model choice and routing |
  61. | **Enterprise clouds** | Azure OpenAI, AWS Bedrock, Vertex AI | Compliance, governance |
  62. | **Specialty** | xAI, Perplexity, Mistral, Cohere, DeepSeek, Hugging Face, Replicate, NVIDIA NIM, MiniMax | Domain-specific |
  63. | **Local runtimes** | Ollama, LM Studio, vLLM, llama.cpp | Privacy, cost control, offline dev |
  64. If you want a governed OpenAI-compatible control plane in front of those providers, [Tuning Engines](https://www.tuningengines.com) is a useful runtime option for policy enforcement, approval gates, MCP and agent tracing, and usage or cost visibility without changing the surrounding agent framework.
  65. Quick start:
  66. ```python
  67. from utilities import get_provider
  68. from utilities.provider_router import ProviderRouter
  69. # Use any single provider
  70. out = get_provider("groq").chat(
  71. [{"role": "user", "content": "Summarize MCP."}],
  72. model="llama-3.1-8b-instant",
  73. )
  74. # Or route by task class with fallback
  75. router = ProviderRouter()
  76. out = router.chat(messages, task_class="cheap") # Groq → DeepSeek → Together → OpenRouter
  77. ```
  78. 📖 [providers/README.md](providers/README.md) • [providers/provider_matrix.md](providers/provider_matrix.md) • [providers/router_patterns.md](providers/router_patterns.md) • [providers/local_models.md](providers/local_models.md)
  79. ---
  80. ## Repository map
  81. ```
  82. .
  83. ├── README.md • llms.txt • llms-full.txt
  84. ├── agent_os/ ← the Agent OS concept, layers, workspace examples
  85. ├── providers/ ← 24+ provider docs + adapters + router patterns
  86. ├── templates/ ← AGENTS.md / SOUL.md / MEMORY.md / SKILL.md / DESIGN_DOC / ADR / …
  87. ├── skills/ ← design guide + taxonomy + maturity model + curated catalog + 4 examples
  88. ├── memory/ ← memory taxonomy, distillation, security, examples
  89. ├── mcp/ ← MCP basics, architecture, security, server catalog, examples
  90. ├── prompt_engineering/ ← agent prompt patterns, instruction hierarchy, defenses
  91. ├── coding_agents/ ← Claude Code, Cursor, Codex, workflows, prompts, review
  92. ├── design_docs/ ← agent + technical design docs, ADR guide, design.md spec
  93. ├── safety/ ← guardrails, approvals, prompt injection, secure checklist
  94. ├── observability/ ← tracing, spans, cost/latency, dashboards
  95. ├── evals/ ← eval design, regression / tool / memory / MCP / safety / prompt
  96. ├── blueprints/ ← production architectures by use case
  97. ├── examples/ ← end-to-end runnable agent workspaces
  98. ├── checklists/ ← agent design, prod readiness, MCP security, …
  99. ├── llm_wiki/ ← LLM-friendly index, glossary, matrices, wiki pattern
  100. ├── docs/ ← framework comparison, best practices, beginners' guide
  101. ├── tutorials/ ← RAG, memory, fine-tuning, chat-with-X
  102. ├── utilities/ ← LLMProvider + router + provider_config
  103. ├── agents/ ← 100+ curated agent skeletons (preserved)
  104. ├── complete_apps/, web_apps/, notebooks/, datasets/, design/, resources/, scripts/, tests/, ecosystem/
  105. └── .github/ ← issue / PR templates
  106. ```
  107. ---
  108. ## Skills ecosystem
  109. A curated, in-repo catalog plus a clear taxonomy and maturity model:
  110. - [skills/skill_design_guide.md](skills/skill_design_guide.md) — write triggers the model picks
  111. - [skills/skill_vs_tool_vs_mcp.md](skills/skill_vs_tool_vs_mcp.md) — when to use which
  112. - [skills/skill_taxonomy.md](skills/skill_taxonomy.md) — domains, tags, risk
  113. - [skills/skill_maturity_model.md](skills/skill_maturity_model.md) — experimental → production
  114. - [skills/skill_packaging.md](skills/skill_packaging.md) — ship a portable skill
  115. - [skills/skill_validation.md](skills/skill_validation.md) — lint / smoke / eval
  116. - [skills/awesome_skills_catalog.md](skills/awesome_skills_catalog.md) — broader ecosystem map
  117. - [skills/catalog/](skills/catalog/) — index + per-domain skills
  118. - [skills/examples/](skills/examples/) — four full reference skills
  119. Curated skills shipped: research-summarizer, repo-auditor, mcp-security-reviewer, agent-memory-curator, api-design-reviewer, pr-summarizer, adr-writer, incident-postmortem, sprint-planner, dataset-profiler.
  120. ---
  121. ## Prompt engineering
  122. A dedicated section, agent-focused:
  123. - [prompt_engineering/agent_prompt_patterns.md](prompt_engineering/agent_prompt_patterns.md)
  124. - [prompt_engineering/system_prompt_design.md](prompt_engineering/system_prompt_design.md)
  125. - [prompt_engineering/instruction_hierarchy.md](prompt_engineering/instruction_hierarchy.md)
  126. - [prompt_engineering/context_engineering.md](prompt_engineering/context_engineering.md)
  127. - [prompt_engineering/tool_use_prompting.md](prompt_engineering/tool_use_prompting.md)
  128. - [prompt_engineering/planning_and_reflection.md](prompt_engineering/planning_and_reflection.md)
  129. - [prompt_engineering/memory_prompting.md](prompt_engineering/memory_prompting.md)
  130. - [prompt_engineering/prompt_injection_defense.md](prompt_engineering/prompt_injection_defense.md)
  131. - [prompt_engineering/prompt_eval_methods.md](prompt_engineering/prompt_eval_methods.md)
  132. - [prompt_engineering/anti_patterns.md](prompt_engineering/anti_patterns.md)
  133. Templates: [SYSTEM_PROMPT](templates/SYSTEM_PROMPT.md.template), [AGENT_PROMPT](templates/AGENT_PROMPT.md.template). Checklist: [agent_prompt_checklist](checklists/agent_prompt_checklist.md).
  134. ---
  135. ## Use this repo with coding agents
  136. The handbook is *itself* a great surface for coding agents. Drop your favorite tool (Claude Code, Cursor, Codex, Aider, Cline) into the repo:
  137. - [llms.txt](llms.txt) gives the agent an index in 30 seconds
  138. - [coding_agents/](coding_agents/) has tool-specific notes + prompts
  139. - [coding_agents/prompts/](coding_agents/prompts/) — repo audit, modernization, feature, bugfix, provider expansion, docs update, release review
  140. - [templates/CODING_AGENT_TASK.md.template](templates/CODING_AGENT_TASK.md.template) — task contract template
  141. - [templates/REPO_MODERNIZATION_PROMPT.md.template](templates/REPO_MODERNIZATION_PROMPT.md.template) — multi-phase modernization
  142. The guidance is **tool-neutral**: same `AGENTS.md`, same workflows, regardless of harness.
  143. ---
  144. ## Design docs
  145. Agent + technical design docs, ADRs, reviews, rollouts, and the `DESIGN.md` machine-readable spec for design tokens:
  146. - [design_docs/agent_design_doc.md](design_docs/agent_design_doc.md)
  147. - [design_docs/technical_design_doc.md](design_docs/technical_design_doc.md)
  148. - [design_docs/adr_guide.md](design_docs/adr_guide.md)
  149. - [design_docs/design_review.md](design_docs/design_review.md)
  150. - [design_docs/rollout_plan.md](design_docs/rollout_plan.md)
  151. - [design_docs/design_md_spec.md](design_docs/design_md_spec.md)
  152. - [design_docs/examples/](design_docs/examples/) — research / MCP / memory / provider-router worked examples
  153. Templates: [DESIGN_DOC](templates/DESIGN_DOC.md.template), [ADR](templates/ADR.md.template).
  154. ---
  155. ## Frameworks at a glance
  156. | Framework | Best for | Lang | MCP | Tracing |
  157. |---|---|---|---|---|
  158. | OpenAI Agents SDK | Production agents | Py / JS | ✅ | ✅ built-in |
  159. | LangGraph | Stateful, branching graphs | Py / JS | ✅ | ✅ LangSmith |
  160. | CrewAI | Role-based teams | Py | ✅ | ⚠️ via partners |
  161. | AutoGen (AG2) | Event-driven multi-agent + HITL | Py | ⚠️ partial | ✅ |
  162. | LlamaIndex Workflows | Data-heavy / RAG-first | Py / TS | ✅ | ✅ |
  163. | Pydantic AI | Type-safe, FastAPI-native | Py | ✅ | ✅ Logfire |
  164. | Smolagents | Code-execution mini-agents | Py | ⚠️ | basic |
  165. | Semantic Kernel | .NET / enterprise / Azure | C# / Py / Java | ✅ | ✅ |
  166. | DSPy | Programmatic prompt optimization | Py | — | ✅ |
  167. | Strands Agents | Provider-agnostic, OpenTelemetry | Py | ✅ | ✅ OTEL |
  168. | Vercel AI SDK | App-layer agents in Next.js | TS / JS | ✅ | ✅ |
  169. | Google ADK | Gemini / Vertex hierarchical tools | Py | ✅ | ✅ |
  170. 📖 Full comparison + decision tree: [docs/framework_comparison.md](docs/framework_comparison.md). Capability tags hedged: verify against current upstream docs.
  171. ---
  172. ## Skills, MCP, and Memory in one minute
  173. - **Skills** are reusable, model-loaded *workflows* (`SKILL.md` + scripts + references). Use when a task is repeatable, multi-step, and benefits from progressive disclosure. → [skills/](skills/)
  174. - **MCP** (Model Context Protocol) is a *standard* for exposing tools/context to any agent. Use when integrations should be reusable (GitHub, filesystem, browser, internal APIs). → [mcp/](mcp/)
  175. - **Memory** is *durable state* across runs (`MEMORY.md`, vector stores, decision logs). → [memory/](memory/)
  176. A useful rule of thumb:
  177. | If the thing is… | Use |
  178. |---|---|
  179. | A repeatable workflow with steps and references | **Skill** |
  180. | An external system with tools to call | **MCP** server |
  181. | State that should outlive the current run | **Memory** |
  182. | A single function the model needs once | Plain **tool** |
  183. 📖 Decision matrix: [skills/skill_vs_tool_vs_mcp.md](skills/skill_vs_tool_vs_mcp.md)
  184. ---
  185. ## Guardrails & safety
  186. Production agents need risk-tiered tool controls and human approval gates for high-impact actions.
  187. | Risk level | Examples | Approval |
  188. |---|---|---|
  189. | **Low** | read-only search, summarization | none |
  190. | **Medium** | drafting files, creating tickets | sometimes |
  191. | **High** | sending email, modifying repos, running shell | required |
  192. | **Critical** | deleting data, spending money, changing permissions | always + audit |
  193. 📖 [safety/README.md](safety/README.md) • [safety/prompt_injection.md](safety/prompt_injection.md) • [safety/secure_agent_checklist.md](safety/secure_agent_checklist.md)
  194. ---
  195. ## Observability & evals
  196. You cannot ship what you cannot measure. The handbook ships:
  197. - A tracing primer ([observability/tracing.md](observability/tracing.md)) and span model ([observability/spans.md](observability/spans.md))
  198. - Cost / latency / failure analysis playbooks
  199. - Eval design + datasets ([evals/](evals/)): regression, tool-call, memory, MCP, safety, **prompt** evals
  200. - A curated guide to [evaluation_frameworks](evaluation_frameworks/) — Promptfoo, DeepEval, RAGAs, Langfuse, Phoenix, TruLens, LangSmith, MLflow
  201. ---
  202. ## Templates (copy-paste ready)
  203. | File | Purpose |
  204. |---|---|
  205. | [AGENTS.md](templates/AGENTS.md.template) | Repo-specific agent instructions |
  206. | [SOUL.md](templates/SOUL.md.template) | Identity, voice, values, refusal style |
  207. | [MEMORY.md](templates/MEMORY.md.template) | Durable project + user memory index |
  208. | [USER.md](templates/USER.md.template) | User profile and preferences |
  209. | [TOOLS.md](templates/TOOLS.md.template) | Allowed/restricted/approval-gated tools |
  210. | [SKILL.md](templates/SKILL.md.template) | Skill spec with progressive loading |
  211. | [MCP_SERVER.md](templates/MCP_SERVER.md.template) | Documenting an MCP integration |
  212. | [SYSTEM_PROMPT.md](templates/SYSTEM_PROMPT.md.template) | Long-lived system prompt |
  213. | [AGENT_PROMPT.md](templates/AGENT_PROMPT.md.template) | Per-task / per-session prompt |
  214. | [DESIGN_DOC.md](templates/DESIGN_DOC.md.template) | Agent / technical design doc |
  215. | [ADR.md](templates/ADR.md.template) | Architecture Decision Record |
  216. | [EVAL_PLAN.md](templates/EVAL_PLAN.md.template) | What you'll evaluate and how |
  217. | [GUARDRAILS.md](templates/GUARDRAILS.md.template) | Policy, refusals, escalation |
  218. | [HUMAN_APPROVAL_POLICY.md](templates/HUMAN_APPROVAL_POLICY.md.template) | Who approves what |
  219. | [CODING_AGENT_TASK.md](templates/CODING_AGENT_TASK.md.template) | Task contract for coding agents |
  220. | [REPO_MODERNIZATION_PROMPT.md](templates/REPO_MODERNIZATION_PROMPT.md.template) | Multi-phase modernization |
  221. | [AGENT_RELEASE_CHECKLIST.md](templates/AGENT_RELEASE_CHECKLIST.md) | Ship/no-ship gate |
  222. ---
  223. ## Merged knowledge areas (1.0.1)
  224. This release **merged seven external projects** into the handbook. Each was adapted (not bulk-copied) into the structure above:
  225. | Source theme | Lives in |
  226. |---|---|
  227. | Skills catalog + taxonomy patterns | [skills/](skills/) — taxonomy, maturity, packaging, validation, awesome catalog |
  228. | Personal-wiki / self-maintaining KB | [llm_wiki/wiki_pattern.md](llm_wiki/wiki_pattern.md), [docs/llm_readable_docs.md](docs/llm_readable_docs.md) |
  229. | Agent prompt research patterns | [prompt_engineering/](prompt_engineering/) |
  230. | Production coding-agent prompts + workflows | [coding_agents/](coding_agents/) — prompts, workflows, review |
  231. | Machine-readable design specs | [design_docs/design_md_spec.md](design_docs/design_md_spec.md), [templates/DESIGN_DOC.md.template](templates/DESIGN_DOC.md.template) |
  232. | ADRs + design reviews | [design_docs/adr_guide.md](design_docs/adr_guide.md), [design_docs/design_review.md](design_docs/design_review.md) |
  233. 📖 Full migration plan: [MIGRATION_AND_PROVIDER_EXPANSION_PLAN.md](MIGRATION_AND_PROVIDER_EXPANSION_PLAN.md)
  234. ---
  235. ## Supported LLM providers
  236. The [`utilities/llm_provider.py`](utilities/llm_provider.py) module exposes a single `LLMProvider` interface (and a backwards-compatible `complete()` function). Switch via `LLM_PROVIDER` without touching agent code; route automatically with [`ProviderRouter`](utilities/provider_router.py).
  237. 24+ providers across frontier / fast / marketplace / enterprise / specialty / local. See:
  238. - [providers/provider_matrix.md](providers/provider_matrix.md) — capability comparison
  239. - [providers/env_vars.md](providers/env_vars.md) — every variable
  240. - [.env.example](.env.example) — copy-and-fill
  241. ---
  242. ## Contributing
  243. Contributions are very welcome — new examples, framework updates, fixes, and translations all help. Start with:
  244. - [CONTRIBUTING.md](CONTRIBUTING.md) — workflow, scope, quality bar
  245. - [.github/PULL_REQUEST_TEMPLATE.md](.github/PULL_REQUEST_TEMPLATE.md)
  246. - [.github/ISSUE_TEMPLATE/](.github/ISSUE_TEMPLATE/)
  247. - [checklists/open_source_quality_checklist.md](checklists/open_source_quality_checklist.md)
  248. ## Roadmap & changelog
  249. - [ROADMAP.md](ROADMAP.md) — what's next
  250. - [CHANGELOG.md](CHANGELOG.md) — what shipped
  251. ## License
  252. MIT — see [LICENSE](LICENSE).
  253. ## Maintainer
  254. Curated & maintained by **Sayed Allam** ([oxbshw](https://github.com/oxbshw)). If this handbook helped you ship, please ⭐ the repo and open a PR with what *you* learned along the way.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.