Skip to content
Back to search

actlens

AI agent that answers EU AI Act questions from the official text, with verified citations, a live trace, MCP server and evals

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/itsmohitshekhawat--actlens/badge.svg)](/listings/itsmohitshekhawat--actlens)

HTML

<a href="/listings/itsmohitshekhawat--actlens"><img src="/api/listings/itsmohitshekhawat--actlens/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 90 lines
  1. # ActLens
  2. **An AI agent that answers questions about the EU AI Act from the official text, cites every claim, and checks its own citations.**
  3. Live demo: **https://actlens.mohitshekhawat.com** · Evaluation: [/eval](https://actlens.mohitshekhawat.com/eval) · MCP server: `https://actlens.mohitshekhawat.com/mcp`
  4. Plain RAG searches once and hopes the right passage is in the results. ActLens is *agentic*: the model decides what to search, reads whole articles when a snippet is not enough, searches again with new words, and only then answers. A deterministic checker then matches every citation against the passages the agent actually retrieved.
  5. ## Results (50 hand-written questions, `data/evalset.json`)
  6. | | Result |
  7. |---|---|
  8. | Agent cites a correct article | **100%** (50/50) |
  9. | One search with the raw question (plain RAG) has a correct article in its top 6 | 78% |
  10. | Answers where every citation is verified | 98% (the checker flagged one citation the agent never read) |
  11. | Average cost / median time per answer | $0.0008 / 6 s (DeepSeek V4.1 Flash, 3.6 model calls) |
  12. Search alone (no model), article-level:
  13. | Method | hit@1 | hit@5 | MRR |
  14. |---|---|---|---|
  15. | Keyword (BM25) | 0.56 | **0.84** | 0.655 |
  16. | Meaning (static embeddings) | 0.54 | 0.82 | 0.655 |
  17. | Hybrid (RRF, recitals ×0.7) | **0.66** | 0.80 | **0.712** |
  18. Hybrid puts the right article first most often, which matters most for an agent that searches several times. The recital weight was tuned on the same 50 questions, so treat these numbers as optimistic.
  19. ## How it works
  20. ```
  21. official XHTML ──► ingest.py ──► 800 passages (article paragraphs, single definitions, annex parts, recitals)
  22. │
  23. build_index.py ├─► BM25 keyword index
  24. └─► Model2Vec static embeddings (30 MB, no PyTorch)
  25. question ──► agent loop (DeepSeek, tool calling, max 6 rounds)
  26. ├─ search_law(query) ─► hybrid search, Reciprocal Rank Fusion
  27. └─ read(ref) ─► full article / annex / recital
  28. ──► answer with [Art. 5(1)] citations ──► citation check ──► streamed to the page (SSE)
  29. ```
  30. - **Data:** Regulation (EU) 2024/1689 from the EU Publications Office API, split by legal structure so every passage has an exact citation label.
  31. - **Search:** `bm25s` + `model2vec`, merged with RRF (`score = Σ 1/(60 + rank)`). Under 1 ms per query on a CPU.
  32. - **Agent:** OpenAI-compatible tool calling (`app/agent.py`). Every step is streamed to the browser as Server-Sent Events, so you watch it work.
  33. - **Citation check:** no extra model call. A citation counts as verified only if the agent retrieved that article, annex or recital.
  34. - **MCP:** the same tools are exposed through `fastmcp` at `/mcp` (`search_ai_act`, `read_ai_act`, `ask_actlens`).
  35. - **Cost control:** a daily USD budget and a per-visitor daily limit in SQLite (only hashed IPs are stored). Search and the eval page stay free when the budget is used up.
  36. - **Footprint:** about 150 MB RAM in Docker, so it runs on a small VPS.
  37. ## Run it
  38. ```bash
  39. python -m venv .venv && . .venv/bin/activate
  40. pip install -r requirements-dev.txt
  41. python scripts/build_index.py # builds data/index/
  42. cp .env.example .env # add LLM_API_KEY
  43. export $(grep -v '^#' .env | xargs)
  44. uvicorn app.main:app --reload # http://localhost:8000
  45. pytest -q # tests, no API key needed
  46. python scripts/eval_retrieval.py # search metrics
  47. python scripts/eval_agent.py # agent metrics (about $0.04 per run)
  48. ```
  49. Or with Docker: `docker build -t actlens . && docker run -p 8000:8000 -e LLM_API_KEY=... actlens`.
  50. Use it from an MCP client:
  51. ```json
  52. { "mcpServers": { "actlens": { "type": "http", "url": "https://actlens.mohitshekhawat.com/mcp" } } }
  53. ```
  54. ## Project layout
  55. | Path | What it does |
  56. |---|---|
  57. | `scripts/ingest.py` | Official XHTML → `data/corpus.json` (800 labelled passages) |
  58. | `scripts/build_index.py` | BM25 + embedding index |
  59. | `app/search.py` | Hybrid search, RRF, reference lookup |
  60. | `app/agent.py` | Agent loop, tools, citation checker |
  61. | `app/llm.py` | Model client (any OpenAI-compatible API), cost tracking |
  62. | `app/budget.py` | Daily budget + per-visitor limits |
  63. | `app/mcp_server.py` | MCP tools |
  64. | `app/main.py` | FastAPI routes, SSE streaming |
  65. | `scripts/eval_*.py`, `data/evalset.json` | Evaluation |
  66. ## Limits and next steps
  67. - The eval set is small (50 questions) and was written by the author. A larger, independently written set would be fairer.
  68. - Next: a cross-encoder reranker; guidelines and harmonised standards as extra sources; an LLM judge for faithfulness, checked against human labels.
  69. ActLens explains the text; it is not legal advice. Law text © European Union, reused under Commission Decision 2011/833/EU. Code: MIT licence.

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.