Skip to content
Back to search

custom-agent-with-skills

Pydantic AI agent that implements the idea of Claude Skills (progressive disclosure) with no reliance on Claude itself.

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/coleam00--custom-agent-with-skills/badge.svg)](/listings/coleam00--custom-agent-with-skills)

HTML

<a href="/listings/coleam00--custom-agent-with-skills"><img src="/api/listings/coleam00--custom-agent-with-skills/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

These steps run a script from the network that findsafeskills does not scan.

Prerequisites

- Python 3.11 or higher - [UV](https://docs.astral.sh/uv/) - Fast Python package manager - An API key for an LLM provider (OpenRouter, OpenAI, Anthropic, etc.)

Installation

1. **Install UV** (if not already installed):

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

2. **Clone the repository**:

git clone https://github.com/coliam00/custom-agent-with-skills.git
cd custom-agent-with-skills

3. **Sync dependencies**:

uv sync

This creates a virtual environment and installs all dependencies automatically.

4. **Configure environment variables**:

Copy `.env.example` to `.env` and set your API key:

cp .env.example .env

Edit `.env` with your API credentials:

# LLM Provider Configuration
LLM_PROVIDER=openrouter
LLM_API_KEY=sk-or-v1-your-api-key-here
LLM_MODEL=anthropic/claude-haiku-4.5
LLM_BASE_URL=https://openrouter.ai/api/v1

Running the Agent

Start the CLI agent:

uv run python -m src.cli

You'll see a Rich-based interface where you can interact with the agent. Try asking: - "What's the weather in New York?" - "Find papers about machine learning" - "What can I make with chicken and garlic?" - "What time is it in Tokyo?"

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 444 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. # Custom Skill-Based Pydantic AI Agent
  2. A framework-agnostic skill system for AI agents that implements **progressive disclosure** - a powerful pattern for managing context efficiently. This project demonstrates how to extract successful patterns from proprietary AI systems (like Claude Skills) and implement them in open frameworks like Pydantic AI.
  3. ## Key Features
  4. - **Progressive Disclosure**: Skills load instructions and resources on-demand, eliminating context window constraints
  5. - **Framework Agnostic**: Skills work with any AI framework, not locked to a specific vendor
  6. - **Type Safe**: Full Pydantic models and type hints throughout
  7. - **Extensible**: Easy to add new skills by following a simple directory structure
  8. - **Production Ready**: Includes comprehensive testing patterns and examples
  9. ## What is Progressive Disclosure?
  10. Progressive disclosure is a technique for managing AI context efficiently. Instead of loading all possible instructions into the system prompt, skills are loaded in three levels:
  11. ```
  12. Level 1: Metadata (~100 tokens per skill)
  13. - Name and brief description in system prompt
  14. - Agent decides which skill might be relevant
  15. Level 2: Full Instructions (loaded on-demand)
  16. - Complete SKILL.md with detailed instructions
  17. - Only loaded when agent chooses to use the skill
  18. Level 3: Resources (loaded on-demand)
  19. - Reference files, scripts, and other resources
  20. - Only loaded when instructions reference them
  21. ```
  22. This pattern allows an agent to have access to potentially hundreds of skills without overwhelming the context window.
  23. ## Quick Start
  24. ### Prerequisites
  25. - Python 3.11 or higher
  26. - [UV](https://docs.astral.sh/uv/) - Fast Python package manager
  27. - An API key for an LLM provider (OpenRouter, OpenAI, Anthropic, etc.)
  28. ### Installation
  29. 1. **Install UV** (if not already installed):
  30. ```bash
  31. # macOS/Linux
  32. curl -LsSf https://astral.sh/uv/install.sh | sh
  33. # Windows
  34. powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
  35. ```
  36. 2. **Clone the repository**:
  37. ```bash
  38. git clone https://github.com/coliam00/custom-agent-with-skills.git
  39. cd custom-agent-with-skills
  40. ```
  41. 3. **Sync dependencies**:
  42. ```bash
  43. uv sync
  44. ```
  45. This creates a virtual environment and installs all dependencies automatically.
  46. 4. **Configure environment variables**:
  47. Copy `.env.example` to `.env` and set your API key:
  48. ```bash
  49. cp .env.example .env
  50. ```
  51. Edit `.env` with your API credentials:
  52. ```bash
  53. # LLM Provider Configuration
  54. LLM_PROVIDER=openrouter
  55. LLM_API_KEY=sk-or-v1-your-api-key-here
  56. LLM_MODEL=anthropic/claude-haiku-4.5
  57. LLM_BASE_URL=https://openrouter.ai/api/v1
  58. ```
  59. ### Running the Agent
  60. Start the CLI agent:
  61. ```bash
  62. uv run python -m src.cli
  63. ```
  64. You'll see a Rich-based interface where you can interact with the agent. Try asking:
  65. - "What's the weather in New York?"
  66. - "Find papers about machine learning"
  67. - "What can I make with chicken and garlic?"
  68. - "What time is it in Tokyo?"
  69. ## Included Skills
  70. ### Weather Skill
  71. Get current weather and forecasts for any location using the Open-Meteo API.
  72. - **Level 2**: Instructions for getting weather data with coordinate lookup
  73. - **Level 3**: Complete API reference documentation
  74. ```
  75. User: What's the weather in Tokyo?
  76. Agent: [loads weather skill] [loads API reference] The current weather in Tokyo is...
  77. ```
  78. ### Research Assistant Skill
  79. Search academic papers using the Semantic Scholar API (214M+ papers, free).
  80. - **Level 2**: Instructions for searching papers, authors, and citations
  81. - **Level 3**: API reference + search tips guide
  82. ```
  83. User: Find papers about transformer neural networks
  84. Agent: [loads research_assistant skill] [loads api_reference]
  85. Here are the most cited papers on transformers...
  86. ```
  87. ### Recipe Finder Skill
  88. Find recipes by ingredients, dietary needs, or cuisine using TheMealDB and Spoonacular APIs.
  89. - **Level 2**: Instructions for recipe search and meal planning
  90. - **Level 3**: API reference + comprehensive dietary restrictions guide
  91. ```
  92. User: What can I make with chicken and garlic?
  93. Agent: [loads recipe_finder skill] [loads dietary_guide]
  94. Here are some delicious recipes you can make...
  95. ```
  96. ### World Clock Skill
  97. Get current time in any timezone and convert times between locations.
  98. - **Level 2**: Instructions with 40+ pre-mapped cities to timezone IDs
  99. ```
  100. User: What time is it in London?
  101. Agent: [loads world_clock skill] It's currently 3:30 PM GMT in London...
  102. ```
  103. ### Code Review Skill
  104. Review code for quality, security, and best practices with extensive reference documentation (~45KB).
  105. - **Level 2**: Multi-step review workflow
  106. - **Level 3**: Three reference files for comprehensive code analysis
  107. ```
  108. User: Review this code for security issues
  109. Agent: [loads code_review skill] [loads security_checklist]
  110. I'll analyze this against security best practices...
  111. ```
  112. ## Architecture
  113. ```
  114. custom-agent-with-skills/
  115. |-- src/ # Core agent implementation
  116. | |-- agent.py # Pydantic AI agent with skill tools
  117. | |-- skill_loader.py # Skill discovery and metadata parsing
  118. | |-- skill_tools.py # Progressive disclosure tools
  119. | |-- skill_toolset.py # FunctionToolset for reusable tools
  120. | |-- http_tools.py # HTTP request tools
  121. | |-- dependencies.py # Agent dependencies
  122. | |-- providers.py # LLM provider configuration
  123. | |-- settings.py # Pydantic Settings configuration
  124. | |-- prompts.py # System prompt templates
  125. | +-- cli.py # Rich-based CLI interface
  126. |
  127. |-- skills/ # Skill library (5 skills)
  128. | |-- weather/ # Weather forecasts via Open-Meteo
  129. | | |-- SKILL.md
  130. | | +-- references/
  131. | | +-- api_reference.md
  132. | |
  133. | |-- research_assistant/ # Academic paper search via Semantic Scholar
  134. | | |-- SKILL.md
  135. | | +-- references/
  136. | | |-- api_reference.md
  137. | | +-- search_tips.md
  138. | |
  139. | |-- recipe_finder/ # Recipe search via TheMealDB/Spoonacular
  140. | | |-- SKILL.md
  141. | | +-- references/
  142. | | |-- api_reference.md
  143. | | +-- dietary_guide.md
  144. | |
  145. | |-- world_clock/ # Timezone conversions via WorldTimeAPI
  146. | | +-- SKILL.md
  147. | |
  148. | +-- code_review/ # Code analysis with extensive docs (~45KB)
  149. | |-- SKILL.md
  150. | |-- references/
  151. | | |-- best_practices.md
  152. | | |-- security_checklist.md
  153. | | +-- common_antipatterns.md
  154. | +-- scripts/
  155. | +-- lint_patterns.py
  156. |
  157. |-- tests/ # Test suite
  158. | |-- test_skill_loader.py # Skill loader tests
  159. | |-- test_skill_tools.py # Tool tests
  160. | |-- test_agent.py # Agent integration tests (71 tests)
  161. | +-- evals/ # Evaluation system
  162. | |-- skill_loading.yaml
  163. | |-- response_quality.yaml
  164. | |-- new_skills.yaml
  165. | |-- evaluators.py
  166. | +-- run_evals.py
  167. |
  168. |-- scripts/ # Validation and test scripts
  169. | |-- validate_skills.py
  170. | |-- test_agent.py
  171. | +-- run_full_validation.py
  172. |
  173. +-- examples/ # Reference implementations
  174. ```
  175. ## Creating Your Own Skill
  176. ### 1. Create the Skill Directory
  177. ```bash
  178. mkdir -p skills/my_skill/references
  179. ```
  180. ### 2. Create SKILL.md
  181. Every skill must have a `SKILL.md` file with YAML frontmatter:
  182. ```markdown
  183. ---
  184. name: my_skill
  185. description: Brief description for agent discovery (1-2 sentences)
  186. version: 1.0.0
  187. author: Your Name
  188. ---
  189. # My Skill
  190. Detailed description of what this skill does.
  191. ## When to Use
  192. - Scenario 1
  193. - Scenario 2
  194. ## Available Operations
  195. 1. Operation 1: Description
  196. 2. Operation 2: Description
  197. ## Instructions
  198. Step-by-step instructions for using this skill...
  199. ## Resources
  200. - `references/guide.md` - Detailed guide
  201. - `scripts/helper.py` - Helper script
  202. ## Examples
  203. ### Example 1
  204. User asks: "..."
  205. Response approach: ...
  206. ```
  207. ### 3. Add Reference Files (Optional)
  208. Place any supporting documentation in the `references/` directory:
  209. ```markdown
  210. # skills/my_skill/references/guide.md
  211. Detailed documentation that the agent can load when needed...
  212. ```
  213. ### 4. Test Your Skill
  214. Verify your skill is discovered:
  215. ```python
  216. from src.skill_loader import SkillLoader
  217. from pathlib import Path
  218. loader = SkillLoader(Path("skills"))
  219. skills = loader.discover_skills()
  220. for skill in skills:
  221. print(f"- {skill.name}: {skill.description}")
  222. ```
  223. ## How the Agent Uses Skills
  224. ### System Prompt (Level 1)
  225. The agent's system prompt includes a summary of all available skills:
  226. ```
  227. Available Skills:
  228. - **weather**: Get weather information for locations using Open-Meteo API.
  229. - **research_assistant**: Search academic papers using Semantic Scholar.
  230. - **recipe_finder**: Search recipes by ingredients, cuisine, or dietary needs.
  231. - **world_clock**: Get current time in any timezone and convert times.
  232. - **code_review**: Review code for quality, security, and best practices.
  233. ```
  234. ### Loading Skills (Level 2)
  235. When the agent decides a skill is relevant, it calls `load_skill_tool`:
  236. ```python
  237. # Agent automatically calls this when needed
  238. instructions = await load_skill_tool(ctx, skill_name="weather")
  239. # Returns: Full SKILL.md content (without frontmatter)
  240. ```
  241. ### Loading Resources (Level 3)
  242. When instructions reference a file, the agent calls `read_skill_file_tool`:
  243. ```python
  244. # Agent loads specific resources as needed
  245. api_docs = await read_skill_file_tool(
  246. ctx,
  247. skill_name="weather",
  248. file_path="references/api_reference.md"
  249. )
  250. ```
  251. ## Available Tools
  252. The agent has access to these tools:
  253. | Tool | Description |
  254. |------|-------------|
  255. | `load_skill_tool` | Load full instructions for a skill (Level 2) |
  256. | `read_skill_file_tool` | Read a file from a skill's directory (Level 3) |
  257. | `list_skill_files_tool` | List available files in a skill |
  258. | `http_get_tool` | Make HTTP GET requests |
  259. | `http_post_tool` | Make HTTP POST requests |
  260. ## Switching LLM Providers
  261. The agent supports three LLM providers. Configure via `.env`:
  262. ### OpenRouter (Recommended)
  263. ```bash
  264. LLM_PROVIDER=openrouter
  265. LLM_API_KEY=sk-or-v1-your-key
  266. LLM_MODEL=anthropic/claude-haiku-4.5
  267. LLM_BASE_URL=https://openrouter.ai/api/v1
  268. # Optional: App attribution
  269. OPENROUTER_APP_URL=https://yourdomain.com
  270. OPENROUTER_APP_TITLE=Your App Name
  271. ```
  272. ### OpenAI
  273. ```bash
  274. LLM_PROVIDER=openai
  275. LLM_API_KEY=sk-proj-your-openai-api-key
  276. LLM_MODEL=gpt-4o-mini
  277. # LLM_BASE_URL not needed for OpenAI
  278. ```
  279. ### Ollama (Local)
  280. ```bash
  281. LLM_PROVIDER=ollama
  282. LLM_API_KEY=ollama
  283. LLM_MODEL=llama3.2
  284. LLM_BASE_URL=http://localhost:11434/v1
  285. ```
  286. ## Testing & Validation
  287. ### Running Tests
  288. ```bash
  289. # Unit tests
  290. uv run pytest tests/test_skill_loader.py -v
  291. uv run pytest tests/test_skill_tools.py -v
  292. # Integration tests
  293. uv run pytest tests/test_agent.py -v
  294. # All tests
  295. uv run pytest tests/ -v
  296. ```
  297. ### Running Evaluations
  298. The project includes a comprehensive evaluation system with YAML-based datasets and custom evaluators (25 test cases across all 5 skills):
  299. ```bash
  300. # Run all evals (25 test cases)
  301. uv run python -m tests.evals.run_evals
  302. # Run specific dataset
  303. uv run python -m tests.evals.run_evals --dataset skill_loading
  304. uv run python -m tests.evals.run_evals --dataset new_skills
  305. uv run python -m tests.evals.run_evals --dataset response_quality
  306. # Verbose output with reasons
  307. uv run python -m tests.evals.run_evals --verbose
  308. ```
  309. ### Validation Scripts
  310. ```bash
  311. # Test agent interactively with predefined queries
  312. uv run python -m scripts.test_agent
  313. # Validate skill structure and content
  314. uv run python -m scripts.validate_skills
  315. # Run full validation pipeline
  316. uv run python -m scripts.run_full_validation
  317. ```
  318. ## Observability with Logfire (Optional)
  319. Enable Logfire for production monitoring and debugging:
  320. 1. Get Logfire token: `logfire auth`
  321. 2. Set in `.env`: `LOGFIRE_TOKEN=your-token`
  322. 3. Run agent - traces appear at https://logfire.pydantic.dev
  323. Without token, Logfire is disabled and agent works normally.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.