Skip to content
Back to search

dingo

SkillWorks with Claude CodeGitHub

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026. Its scripts were only partly scanned.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/migoxlab/dingo with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/migoxlab/dingo'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/migoxlab--dingo/badge.svg)](/listings/migoxlab--dingo)

HTML

<a href="/listings/migoxlab--dingo"><img src="/api/listings/migoxlab--dingo/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Installation

# Core package (includes rule evaluation, LLM evaluation, MCP server, datasource support)
pip install dingo-python

# With HHEM hallucination detection model (requires transformers + torch)
pip install "dingo-python[hhem]"

# With retrieval benchmark evaluation (MTEB + pytrec-eval-terrier)
pip install "dingo-python[retrieval]"

# With all features (HHEM + Agent + Retrieval)
pip install "dingo-python[all]"

Example Use Cases of Dingo

1. Evaluate LLM chat data

from dingo.config.input_args import EvaluatorLLMArgs
from dingo.io.input import Data
from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
from dingo.model.rule.rule_common import RuleSpecialCharacter

data = Data(
    data_id='123',
    prompt="hello, introduce the world",
    content="�I am 8 years old. ^I love apple because:"
)

def llm():
    LLMTextQualityV4.dynamic_config = EvaluatorLLMArgs(
        key='YOUR_API_KEY',
        api_url='https://api.openai.com/v1/chat/completions',
        model='gpt-4o',
    )
    res = LLMTextQualityV4.eval(data)
    print(res)

def rule():
    res = RuleSpecialCharacter().eval(data)
    print(res)

rule()

2. Evaluate Dataset

from dingo.config import InputArgs
from dingo.exec import Executor

# Evaluate a dataset from Hugging Face
if __name__ == '__main__':
    input_data = {
        "input_path": "tatsu-lab/alpaca",  # Dataset from Hugging Face
        "dataset": {
            "source": "hugging_face",
            "format": "plaintext"  # Format: plaintext
        },
        "executor": {
            "result_save": {
                "bad": True  # Save evaluation results
            }
        },
        "evaluator": [
            {
                "evals": [
                    {"name": "RuleColonEnd"},
                    {"name": "RuleSpecialCharacter"}
                ]
            }
        ]
    }

    input_args = InputArgs(**input_data)
    executor = Executor.exec_map["local"](input_args)
    result = executor.execute()
    print(result)

Command Line Interface

Evaluate with Rule Sets

dingo eval --input .github/env/local_plaintext.json

Evaluate with LLM (e.g., GPT-4o)

dingo eval --input .github/env/local_json.json

---

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 1 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. clawhub/SKILL.md
README.md · README · 797 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

  1. <!-- SEO Meta Information and Structured Data -->
  2. <div itemscope itemtype="https://schema.org/SoftwareApplication" align="center" xmlns="http://www.w3.org/1999/html">
  3. <meta itemprop="name" content="Dingo: A Comprehensive AI Data Quality Evaluation Tool">
  4. <meta itemprop="description" content="Comprehensive AI-powered data quality assessment platform for machine learning datasets, LLM training data validation, hallucination detection, and RAG system evaluation">
  5. <meta itemprop="applicationCategory" content="Data Quality Software">
  6. <meta itemprop="operatingSystem" content="Cross-platform">
  7. <meta itemprop="programmingLanguage" content="Python">
  8. <meta itemprop="url" content="https://github.com/MigoXLab/dingo">
  9. <meta itemprop="downloadUrl" content="https://pypi.org/project/dingo-python/">
  10. <meta itemprop="softwareVersion" content="latest">
  11. <meta itemprop="license" content="Apache-2.0">
  12. <!-- logo -->
  13. <p align="center">
  14. <img src="docs/assets/dingo-logo.png" width="300px" style="vertical-align:middle;" alt="Dingo AI Data Quality Evaluation Tool Logo">
  15. </p>
  16. <!-- badges -->
  17. <p align="center">
  18. <a href="https://github.com/pre-commit/pre-commit"><img src="https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white" alt="pre-commit"></a>
  19. <a href="https://pypi.org/project/dingo-python/"><img src="https://img.shields.io/pypi/v/dingo-python.svg" alt="PyPI version"></a>
  20. <a href="https://pypi.org/project/dingo-python/"><img src="https://img.shields.io/pypi/pyversions/dingo-python.svg" alt="Python versions"></a>
  21. <a href="https://github.com/MigoXLab/dingo/blob/main/LICENSE"><img src="https://img.shields.io/github/license/DataEval/dingo" alt="License"></a>
  22. <a href="https://github.com/MigoXLab/dingo/stargazers"><img src="https://img.shields.io/github/stars/DataEval/dingo" alt="GitHub stars"></a>
  23. <a href="https://github.com/MigoXLab/dingo/network/members"><img src="https://img.shields.io/github/forks/DataEval/dingo" alt="GitHub forks"></a>
  24. <a href="https://github.com/MigoXLab/dingo/issues"><img src="https://img.shields.io/github/issues/DataEval/dingo" alt="GitHub issues"></a>
  25. <a href="https://mseep.ai/app/dataeval-dingo"><img src="https://mseep.net/pr/dataeval-dingo-badge.png" alt="MseeP.ai Security Assessment Badge" height="20"></a>
  26. <a href="https://deepwiki.com/MigoXLab/dingo"><img src="https://deepwiki.com/badge.svg" alt="Ask DeepWiki"></a>
  27. <a href="https://archestra.ai/mcp-catalog/dataeval__dingo"><img src="https://archestra.ai/mcp-catalog/api/badge/quality/DataEval/dingo" alt="Trust Score"></a>
  28. <a href="https://clawhub.ai/e06084/dingo"><img src="https://img.shields.io/badge/ClawHub-Skill-orange?logo=data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCI+PHRleHQgeT0iMTgiIGZvbnQtc2l6ZT0iMTYiPvCfpp48L3RleHQ+PC9zdmc+" alt="ClawHub Skill"></a>
  29. <a href="https://mcptoplist.com/server/glama%2FMigoXLab%2Fdingo"><img src="https://mcptoplist.com/badge/glama%2FMigoXLab%2Fdingo.svg" alt="MCP Toplist"></a>
  30. </p>
  31. </div>
  32. <div align="center">
  33. [English](README.md) · [简体中文](README_zh-CN.md) · [日本語](README_ja.md)
  34. </div>
  35. <!-- join us -->
  36. <p align="center">
  37. 👋 join us on <a href="https://discord.gg/Jhgb2eKWh8" target="_blank">Discord</a> and <a href="./docs/assets/wechat.jpg" target="_blank">WeChat</a>
  38. </p>
  39. <p align="center">
  40. If you like Dingo, please give us a ⭐ on GitHub!
  41. <br/>
  42. <a href="https://github.com/MigoXLab/dingo/stargazers" target="_blank">
  43. <img src="docs/assets/clickstar_2.gif" alt="Click Star" width="480">
  44. </a>
  45. </p>
  46. # Introduction
  47. **Dingo is A Comprehensive AI Data, Model and Application Quality Evaluation Tool**, designed for ML practitioners, data engineers, and AI researchers. It helps you systematically assess and improve the quality of training data, fine-tuning datasets, and production AI systems.
  48. ---
  49. ## 🚀 Enterprise Dingo SaaS Version
  50. Need a **production-grade data quality platform**? Try [Dingo SaaS](https://dingo.openxlab.org.cn) Enterprise Edition!
  51. ### ✨ Compared to the open-source version, SaaS provides:
  52. - 🌐 **Web UI** - Visual evaluation interface, no coding required
  53. - 🔐 **Access Control** - JWT + Google OAuth 2.0
  54. - 📊 **Visual Reports** - Interactive charts, trend analysis, export features
  55. - 🔌 **RESTful API** - Seamless integration with existing systems
  56. <details>
  57. <summary><strong>SaaS — Dashboard</strong></summary>
  58. <img src="docs/assets/saas_dashboard.png" alt="Dingo SaaS dashboard" />
  59. </details>
  60. <details>
  61. <summary><strong>SaaS — Playground</strong></summary>
  62. <img src="docs/assets/saas_playground.png" alt="Dingo SaaS playground" />
  63. </details>
  64. <details>
  65. <summary><strong>SaaS — Evaluation output</strong></summary>
  66. <img src="docs/assets/saas_output.png" alt="Dingo SaaS evaluation output" />
  67. </details>
  68. ### 📝 How to Get Free SaaS Code
  69. 👉 **[Apply for Dingo SaaS Repository Access](https://aicarrier.feishu.cn/share/base/form/shrcn9RqYttByQ5H1np6Yrnmhuf)**
  70. Review time: 1-5 business days | Suitable for enterprise data governance, team collaboration
  71. ---
  72. ## Why Dingo?
  73. 🎯 **Production-Grade Quality Checks** - From pre-training datasets to RAG systems, ensure your AI gets high-quality data
  74. 🗄️ **Multi-Source Data Integration** - Seamlessly connect to Local files, SQL databases (PostgreSQL/MySQL/SQLite), HuggingFace datasets, and S3 storage
  75. 🔍 **Multi-Field Evaluation** - Apply different quality rules to different fields in parallel (e.g., ISBN validation for `isbn`, text quality for `title`)
  76. 🤖 **RAG System Assessment** - Comprehensive evaluation of retrieval and generation quality with 5 academic-backed metrics
  77. 🧠 **LLM & Rule & Agent Hybrid** - Combine fast heuristic rules (30+ built-in) with LLM-based deep assessment
  78. 🚀 **Flexible Execution** - Run locally for rapid iteration or scale with Spark for billion-scale datasets
  79. 📊 **Rich Reporting** - Detailed quality reports with GUI visualization and field-level insights
  80. ## Architecture Diagram
  81. ![Architecture of dingo](./docs/assets/architeture.png)
  82. # Quick Start
  83. ## Installation
  84. ```shell
  85. # Core package (includes rule evaluation, LLM evaluation, MCP server, datasource support)
  86. pip install dingo-python
  87. # With HHEM hallucination detection model (requires transformers + torch)
  88. pip install "dingo-python[hhem]"
  89. # With retrieval benchmark evaluation (MTEB + pytrec-eval-terrier)
  90. pip install "dingo-python[retrieval]"
  91. # With all features (HHEM + Agent + Retrieval)
  92. pip install "dingo-python[all]"
  93. ```
  94. ## Example Use Cases of Dingo
  95. ### 1. Evaluate LLM chat data
  96. ```python
  97. from dingo.config.input_args import EvaluatorLLMArgs
  98. from dingo.io.input import Data
  99. from dingo.model.llm.text_quality.llm_text_quality_v4 import LLMTextQualityV4
  100. from dingo.model.rule.rule_common import RuleSpecialCharacter
  101. data = Data(
  102. data_id='123',
  103. prompt="hello, introduce the world",
  104. content="�I am 8 years old. ^I love apple because:"
  105. )
  106. def llm():
  107. LLMTextQualityV4.dynamic_config = EvaluatorLLMArgs(
  108. key='YOUR_API_KEY',
  109. api_url='https://api.openai.com/v1/chat/completions',
  110. model='gpt-4o',
  111. )
  112. res = LLMTextQualityV4.eval(data)
  113. print(res)
  114. def rule():
  115. res = RuleSpecialCharacter().eval(data)
  116. print(res)
  117. rule()
  118. ```
  119. ### 2. Evaluate Dataset
  120. ```python
  121. from dingo.config import InputArgs
  122. from dingo.exec import Executor
  123. # Evaluate a dataset from Hugging Face
  124. if __name__ == '__main__':
  125. input_data = {
  126. "input_path": "tatsu-lab/alpaca", # Dataset from Hugging Face
  127. "dataset": {
  128. "source": "hugging_face",
  129. "format": "plaintext" # Format: plaintext
  130. },
  131. "executor": {
  132. "result_save": {
  133. "bad": True # Save evaluation results
  134. }
  135. },
  136. "evaluator": [
  137. {
  138. "evals": [
  139. {"name": "RuleColonEnd"},
  140. {"name": "RuleSpecialCharacter"}
  141. ]
  142. }
  143. ]
  144. }
  145. input_args = InputArgs(**input_data)
  146. executor = Executor.exec_map["local"](input_args)
  147. result = executor.execute()
  148. print(result)
  149. ```
  150. ## Command Line Interface
  151. ### Evaluate with Rule Sets
  152. ```shell
  153. dingo eval --input .github/env/local_plaintext.json
  154. ```
  155. ### Evaluate with LLM (e.g., GPT-4o)
  156. ```shell
  157. dingo eval --input .github/env/local_json.json
  158. ```
  159. ---
  160. # MCP Server
  161. Dingo includes a built-in Model Context Protocol (MCP) server for AI agent integration:
  162. ```bash
  163. # Start MCP server (SSE transport, default port 8000)
  164. dingo serve
  165. # Custom port
  166. dingo serve --port 9000
  167. # stdio transport (for Claude Desktop)
  168. dingo serve --transport stdio
  169. ```
  170. For detailed setup and Cursor/Claude Desktop integration, see the dedicated documentation:
  171. [English](README_mcp.md) · [简体中文](README_mcp_zh-CN.md) · [日本語](README_mcp_ja.md)
  172. ## Video Demonstration
  173. To help you get started quickly with Dingo MCP, we've created a video walkthrough:
  174. https://github.com/user-attachments/assets/aca26f4c-3f2e-445e-9ef9-9331c4d7a37b
  175. This video demonstrates step-by-step how to use Dingo MCP server with Cursor.
  176. ---
  177. # 📚 Data Quality Metrics
  178. Dingo provides **100+ evaluation metrics** across multiple dimensions, combining rule-based speed with LLM-based depth.
  179. ## Metric Categories
  180. | Category | Examples | Use Case |
  181. |----------|----------|----------|
  182. | **Pretrain Text Quality** | Completeness, Effectiveness, Similarity, Security | LLM pre-training data filtering |
  183. | **SFT Data Quality** | Honest, Helpful, Harmless (3H) | Instruction fine-tuning data |
  184. | **RAG Evaluation** | Faithfulness, Context Precision, Answer Relevancy | RAG system assessment |
  185. | **Hallucination Detection** | HHEM-2.1-Open, Factuality Check | Production AI reliability |
  186. | **Classification** | Topic categorization, Content labeling | Data organization |
  187. | **Multimodal** | Image-text relevance, VLM quality, OCR visual evaluation | Vision-language data |
  188. | **Security** | PII detection, Perspective API toxicity | Privacy and safety |
  189. 📊 **[View Complete Metrics Documentation →](docs/metrics.md)**
  190. 📖 **[RAG Evaluation Guide →](docs/rag_evaluation_metrics.md)** | **[中文版](docs/rag_evaluation_metrics_zh.md)**
  191. 🔍 **[Hallucination Detection Guide →](docs/hallucination_detection_guide.md)** | **[中文版](docs/hallucination_guide.md)**
  192. ✅ **[Factuality Assessment Guide →](docs/factuality_assessment_guide.md)** | **[中文版](docs/factcheck_guide.md)**
  193. 👁️ **[VLM Render Judge Guide →](docs/en/vlm_render_judge_guide.md)** | **[中文版](docs/vlm_render_judge_guide.md)**
  194. Most metrics are backed by academic research to ensure scientific rigor.
  195. ## Quick Metric Usage
  196. ```python
  197. llm_config = {
  198. "model": "gpt-4o",
  199. "key": "YOUR_API_KEY",
  200. "api_url": "https://api.openai.com/v1/chat/completions"
  201. }
  202. input_data = {
  203. "evaluator": [
  204. {
  205. "fields": {"content": "content"},
  206. "evals": [
  207. {"name": "RuleAbnormalChar"}, # Rule-based (fast)
  208. {"name": "LLMTextQualityV5", "config": llm_config} # LLM-based (deep)
  209. ]
  210. }
  211. ]
  212. }
  213. ```
  214. **Customization**: All prompts are defined in `dingo/model/llm/` directory (organized by category: `text_quality/`, `rag/`, `hhh/`, etc.). Extend or modify them for domain-specific requirements.
  215. ---
  216. # 🎓 Key Concepts for Practitioners
  217. ## What Makes Dingo Production-Ready?
  218. ### 1. **Multi-Field Evaluation Pipeline**
  219. Apply different quality checks to different fields in a single pass:
  220. ```python
  221. "evaluator": [
  222. {"fields": {"content": "isbn"}, "evals": [{"name": "RuleIsbn"}]},
  223. {"fields": {"content": "title"}, "evals": [{"name": "RuleAbnormalChar"}]},
  224. {"fields": {"content": "description"}, "evals": [{"name": "LLMTextQualityV5"}]}
  225. ]
  226. ```
  227. **Why It Matters**: Evaluate structured data (like database tables) without writing separate scripts for each field.
  228. ### 2. **Stream Processing for Large Datasets**
  229. SQL datasources use SQLAlchemy's server-side cursors:
  230. ```python
  231. # Handles billions of rows without OOM
  232. for data in dataset.get_data(): # Yields one row at a time
  233. result = evaluator.eval(data)
  234. ```
  235. **Why It Matters**: Process production databases without exporting to intermediate files.
  236. ### 3. **Field Isolation in Memory**
  237. RAG evaluations prevent context bleeding across different field combinations:
  238. ```
  239. outputs/
  240. ├── user_input,response,retrieved_contexts/ # Faithfulness group
  241. └── user_input,response/ # Answer Relevancy group
  242. ```
  243. **Why It Matters**: Accurate metric calculations when evaluating multiple field combinations.
  244. ### 4. **Hybrid Rule-LLM Strategy**
  245. Combine fast rules (100% coverage) with sampled LLM checks (10% coverage):
  246. ```python
  247. "evals": [
  248. {"name": "RuleAbnormalChar"}, # Fast, runs on all data
  249. {"name": "LLMTextQualityV5"} # Expensive, sample if needed
  250. ]
  251. ```
  252. **Why It Matters**: Balance cost and coverage for production-scale evaluation.
  253. ### 5. **Extensibility Through Registration**
  254. Clean plugin architecture for custom rules, prompts, and models:
  255. ```python
  256. @Model.rule_register('QUALITY_BAD_CUSTOM', ['default'])
  257. class MyCustomRule(BaseRule):
  258. @classmethod
  259. def eval(cls, input_data: Data) -> EvalDetail:
  260. # Example: check if content is empty
  261. if not input_data.content:
  262. return EvalDetail(
  263. metric=cls.__name__,
  264. status=True, # Found an issue
  265. label=[f'{cls.metric_type}.{cls.__name__}'],
  266. reason=["Content is empty"]
  267. )
  268. return EvalDetail(
  269. metric=cls.__name__,
  270. status=False, # No issue found
  271. label=['QUALITY_GOOD']
  272. )
  273. ```
  274. **Why It Matters**: Adapt to domain-specific requirements without forking the codebase.
  275. ---
  276. # 🌟 Feature Highlights
  277. ## 📊 Multi-Source Data Integration
  278. **Diverse Data Sources** - Connect to where your data lives
  279. ✅ **Local Files**: JSONL, CSV, TXT, Parquet
  280. ✅ **SQL Databases**: PostgreSQL, MySQL, SQLite, Oracle, SQL Server (with stream processing)
  281. ✅ **Cloud Storage**: S3 and S3-compatible storage
  282. ✅ **ML Platforms**: Direct HuggingFace datasets integration
  283. **Enterprise-Ready SQL Support** - Production database integration
  284. ✅ Memory-efficient streaming for billion-scale datasets
  285. ✅ Connection pooling and automatic resource cleanup
  286. ✅ Complex SQL queries (JOIN, WHERE, aggregations)
  287. ✅ Multiple dialect support with SQLAlchemy
  288. **Multi-Field Quality Checks** - Different rules for different fields
  289. ✅ Parallel evaluation pipelines (e.g., ISBN validation + text quality simultaneously)
  290. ✅ Field aliasing and nested field extraction (`user.profile.name`)
  291. ✅ Independent result reports per field
  292. ✅ ETL pipeline architecture for flexible data transformation
  293. ---
  294. ## 🤖 RAG System Evaluation
  295. **5 Academic-Backed Metrics** - Based on RAGAS, DeepEval, TruLens research
  296. ✅ **Faithfulness**: Answer-context consistency (hallucination detection)
  297. ✅ **Answer Relevancy**: Answer-query alignment
  298. ✅ **Context Precision**: Retrieval precision
  299. ✅ **Context Recall**: Retrieval recall
  300. ✅ **Context Relevancy**: Context-query relevance
  301. **Comprehensive Reporting** - Auto-aggregated statistics
  302. ✅ Average, min, max, standard deviation for each metric
  303. ✅ Field-grouped results
  304. ✅ Batch and single evaluation modes
  305. 📖 **[View RAG Evaluation Guide →](docs/rag_evaluation_metrics_zh.md)**
  306. ---
  307. ## 🧠 Hybrid Evaluation System
  308. **Rule-Based** - Fast, deterministic, cost-effective
  309. ✅ 30+ built-in rules (text quality, format, PII detection)
  310. ✅ Regex, heuristics, statistical checks
  311. ✅ Custom rule registration
  312. **LLM-Based** - Deep semantic understanding
  313. ✅ OpenAI (GPT-4o, GPT-3.5), DeepSeek, Kimi
  314. ✅ Local models (Llama3, Qwen)
  315. ✅ Vision-Language Models (InternVL, Gemini)
  316. ✅ Custom prompt registration
  317. **Agent-Based** - Multi-step reasoning with tools
  318. ✅ Web search integration (Tavily)
  319. ✅ Adaptive context gathering
  320. ✅ Multi-source fact verification
  321. ✅ Custom agent & tool registration
  322. **Extensible Architecture**
  323. ✅ Plugin-based rule/prompt/model registration
  324. ✅ Clean separation of concerns (agents, tools, orchestration)
  325. ✅ Domain-specific customization
  326. ---
  327. ## 🚀 Flexible Execution & Integration
  328. **Multiple Interfaces**
  329. ✅ CLI for quick checks
  330. ✅ Python SDK for integration
  331. ✅ MCP (Model Context Protocol) server for IDEs (Cursor, etc.)
  332. **Scalable Execution**
  333. ✅ Local executor for rapid iteration
  334. ✅ Spark executor for distributed processing
  335. ✅ Configurable concurrency and batching
  336. **Data Sources**
  337. ✅ **Local Files**: JSONL, CSV, TXT, Parquet formats
  338. ✅ **Hugging Face**: Direct integration with HF datasets hub
  339. ✅ **S3 Storage**: AWS S3 and S3-compatible storage
  340. ✅ **SQL Databases**: PostgreSQL, MySQL, SQLite, Oracle, SQL Server (stream processing for large-scale data)
  341. **Modalities**
  342. ✅ Text (chat, documents, code)
  343. ✅ Images (with VLM support)
  344. ✅ Multimodal (text + image consistency)
  345. ---
  346. ## 📈 Rich Reporting & Visualization
  347. **Multi-Level Reports**
  348. ✅ Summary JSON with overall scores
  349. ✅ Field-level breakdown
  350. ✅ Per-rule violation details
  351. ✅ Type and name distribution
  352. **GUI Visualization (via [Dingo SaaS](https://github.com/MigoXLab/dingo-saas))**
  353. ✅ Web UI with interactive data exploration
  354. ✅ Visual reports with trend analysis
  355. ✅ Anomaly tracking
  356. **Metric Aggregation**
  357. ✅ Automatic statistics (avg, min, max, std_dev)
  358. ✅ Field-grouped metrics
  359. ✅ Overall quality score
  360. ---
  361. # 📖 User Guide
  362. ## 🔧 Extensibility
  363. Dingo uses a clean plugin architecture for domain-specific customization:
  364. ### Custom Rule Registration
  365. ```python
  366. from dingo.model import Model
  367. from dingo.model.rule.base import BaseRule
  368. from dingo.io import Data
  369. from dingo.io.output.eval_detail import EvalDetail
  370. @Model.rule_register('QUALITY_BAD_CUSTOM', ['default'])
  371. class DomainSpecificRule(BaseRule):
  372. """Check domain-specific patterns"""
  373. @classmethod
  374. def eval(cls, input_data: Data) -> EvalDetail:
  375. text = input_data.content
  376. # Your custom logic
  377. is_valid = your_validation_logic(text)
  378. return EvalDetail(
  379. metric=cls.__name__,
  380. status=not is_valid, # False = good, True = bad
  381. label=['QUALITY_GOOD' if is_valid else 'QUALITY_BAD_CUSTOM'],
  382. reason=["Validation details..."]
  383. )
  384. ```
  385. ### Custom LLM/Prompt Registration
  386. ```python
  387. from dingo.model import Model
  388. from dingo.model.llm.base_openai import BaseOpenAI
  389. @Model.llm_register('custom_evaluator')
  390. class CustomEvaluator(BaseOpenAI):
  391. """Custom LLM evaluator with specialized prompts"""
  392. _metric_info = {
  393. "metric_name": "CustomEvaluator",
  394. "metric_type": "LLM-Based Quality",
  395. "category": "Custom Category"
  396. }
  397. prompt = """Your custom prompt here..."""
  398. ```
  399. **Examples:**
  400. - [Custom Rules](examples/register/sdk_register_rule.py)
  401. - [Custom Models](examples/register/sdk_register_llm.py)
  402. ### Agent-Based Evaluation with Tools
  403. Dingo supports agent-based evaluators that can use external tools for multi-step reasoning and adaptive context gathering. Two implementation patterns are available:
  404. **Pattern 1: LangChain-Based** (e.g., `AgentFactCheck`)
  405. - Framework-driven with autonomous multi-step reasoning
  406. - Uses LangChain 1.0's `create_agent` with ReAct pattern
  407. - Best for: Complex reasoning tasks, rapid prototyping
  408. - Less code, more declarative
  409. **Pattern 2: Custom Workflow** (e.g., `AgentHallucination`)
  410. - Developer-driven with explicit workflow control
  411. - Manual tool calls and LLM interactions
  412. - Best for: Composing existing evaluators, domain-specific workflows
  413. - Full control, explicit behavior
  414. Both patterns share the same configuration interface and are transparent to users.
  415. **Built-in Agents:**
  416. - `AgentFactCheck`: LangChain-based fact-checking with autonomous search control
  417. - `AgentHallucination`: Custom workflow hallucination detection with adaptive context gathering
  418. - `ArticleFactChecker`: Two-phase article fact-checking — extracts verifiable claims then verifies each in parallel using web search and Arxiv, with configurable concurrency control
  419. **Quick Example:**
  420. ```python
  421. from dingo.io import Data
  422. from dingo.io.output.eval_detail import EvalDetail
  423. from dingo.model import Model
  424. from dingo.model.llm.agent.base_agent import BaseAgent
  425. @Model.llm_register('MyAgent')
  426. class MyAgent(BaseAgent):
  427. """Custom agent with tool support"""
  428. available_tools = ["tavily_search", "my_custom_tool"]
  429. max_iterations = 5
  430. @classmethod
  431. def eval(cls, input_data: Data) -> EvalDetail:
  432. # Use tools for fact-checking
  433. search_result = cls.execute_tool('tavily_search', query=input_data.content)
  434. # Multi-step reasoning with LLM
  435. result = cls.send_messages([...])
  436. return EvalDetail(...)
  437. ```
  438. For detailed guidance on choosing and implementing agent patterns, see [Agent Development Guide](docs/agent_development_guide.md).
  439. **Configuration Example:**
  440. ```json
  441. {
  442. "evaluator": [{
  443. "evals": [{
  444. "name": "AgentHallucination",
  445. "config": {
  446. "key": "openai-api-key",
  447. "model": "gpt-4",
  448. "parameters": {
  449. "agent_config": {
  450. "max_iterations": 5,
  451. "tools": {
  452. "tavily_search": {"api_key": "tavily-key"}
  453. }
  454. }
  455. }
  456. }
  457. }]
  458. }]
  459. }
  460. ```
  461. **Learn More:**
  462. - [Agent Development Guide](docs/agent_development_guide.md) - Comprehensive guide for creating custom agents and tools
  463. - [AgentHallucination Example](examples/agent/agent_hallucination_example.py) - Production agent example
  464. - [AgentFactCheck Example](examples/agent/agent_executor_example.py) - LangChain agent example
  465. - [ArticleFactChecker Example](examples/agent/agent_article_fact_checking_example.py) - Article-scale two-phase fact verification
  466. ## ⚙️ Execution Modes
  467. ### Local Executor (Development & Small-Scale)
  468. ```python
  469. from dingo.config import InputArgs
  470. from dingo.exec import Executor
  471. input_args = InputArgs(**input_data)
  472. executor = Executor.exec_map["local"](input_args)
  473. result = executor.execute()
  474. # Access results
  475. summary = executor.get_summary() # Overall metrics
  476. bad_data = executor.get_bad_info_list() # Quality issues
  477. good_data = executor.get_good_info_list() # High-quality data
  478. ```
  479. **Best For**: Rapid iteration, debugging, datasets < 100K rows
  480. ### Spark Executor (Production & Large-Scale)
  481. ```python
  482. from pyspark.sql import SparkSession
  483. from dingo.exec import Executor
  484. spark = SparkSession.builder.appName("Dingo").getOrCreate()
  485. spark_rdd = spark.sparkContext.parallelize(your_data)
  486. executor = Executor.exec_map["spark"](
  487. input_args,
  488. spark_session=spark,
  489. spark_rdd=spark_rdd
  490. )
  491. result = executor.execute()
  492. ```
  493. **Best For**: Production pipelines, distributed processing, datasets > 1M rows
  494. ### Retrieval Benchmark Executor
  495. Evaluate search APIs against MTEB standard retrieval benchmarks (SciFact, SCIDOCS, etc.) with metrics like NDCG, Recall, and MRR.
  496. **CLI:**
  497. ```bash
  498. pip install dingo-python[retrieval]
  499. dingo eval-retrieval --backend agentic --tasks SciFact \
  500. --api-url https://api.sciverse.space \
  501. --api-token YOUR_TOKEN --limit 100 --max-workers 8
  502. dingo eval-retrieval --backend meta_search --tasks SciFact \
  503. --api-url https://api.sciverse.space \
  504. --api-token YOUR_TOKEN --limit 100 --rate-limit 1.0 \
  505. --freshness-boost MILD \
  506. --filters-json '[{"field": "publication_published_year", "operator": "FILTER_OP_GTE", "value": 2010}]'
  507. dingo eval-retrieval --backend google_scholar --tasks SciFact \
  508. --api-token YOUR_SERPAPI_KEY --limit 100 --rate-limit 1.0
  509. dingo eval-retrieval --backend openalex --tasks SciFact \
  510. --api-token YOUR_OPENALEX_API_KEY --limit 100
  511. dingo eval-retrieval --backend openalex --tasks SciFact \
  512. --api-token YOUR_OPENALEX_API_KEY --limit 50 \
  513. --search-type semantic --rate-limit 1.0 --max-workers 1
  514. ```
  515. **SDK:**
  516. ```python
  517. from dingo.config import InputArgs
  518. from dingo.exec import Executor
  519. input_data = {
  520. "input_path": "SciFact",
  521. "output_path": "outputs/retrieval_eval",
  522. "executor": {
  523. "retrieval": {
  524. "backend": "agentic",
  525. "api_url": "https://api.sciverse.space",
  526. "api_token": "YOUR_TOKEN",
  527. "limit": 100,
  528. "max_workers": 8,
  529. }
  530. },
  531. }
  532. input_args = InputArgs(**input_data)
  533. executor = Executor.exec_map["retrieval"](input_args)
  534. result = executor.execute()
  535. print(result)
  536. ```
  537. **Best For**: Evaluating search/retrieval APIs against academic benchmarks, comparing retrieval backends
  538. ## Evaluation Reports
  539. After evaluation, Dingo generates:
  540. 1. **Summary Report** (`summary.json`): Overall metrics and scores
  541. 2. **Detailed Reports**: Specific issues for each rule violation
  542. Report Description:
  543. 1. **score**: `num_good` / `total`
  544. 2. **type_ratio**: The count of type / total, such as: `QUALITY_BAD_COMPLETENESS` / `total`
  545. Example summary:
  546. ```json
  547. {
  548. "task_id": "d6c922ec-981c-11ef-b723-7c10c9512fac",
  549. "task_name": "dingo",
  550. "eval_group": "default",
  551. "input_path": "test/data/test_local_jsonl.jsonl",
  552. "output_path": "outputs/d6c921ac-981c-11ef-b723-7c10c9512fac",
  553. "create_time": "20241101_144510",
  554. "score": 50.0,
  555. "num_good": 1,
  556. "num_bad": 1,
  557. "total": 2,
  558. "type_ratio": {
  559. "content": {
  560. "QUALITY_BAD_COMPLETENESS.RuleColonEnd": 0.5,
  561. "QUALITY_BAD_RELEVANCE.RuleSpecialCharacter": 0.5
  562. }
  563. }
  564. }
  565. ```
  566. # 🚀 Roadmap & Contributions
  567. ## Future Plans
  568. - [ ] **Agent-as-a-Judge** - Multi-agent debate patterns for bias reduction and complex reasoning
  569. - [ ] **SaaS Platform** - Hosted evaluation service with API access and dashboard
  570. - [ ] **Audio & Video Modalities** - Extend beyond text/image
  571. - [ ] **Diversity Metrics** - Statistical diversity assessment
  572. - [ ] **Real-time Monitoring** - Continuous quality checks in production pipelines
  573. ## Limitations
  574. The current built-in detection rules and model methods primarily focus on common data quality issues. For special evaluation needs, we recommend customizing detection rules.
  575. # Acknowledgments
  576. - [RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data)
  577. - [mlflow](https://github.com/mlflow/mlflow)
  578. - [deepeval](https://github.com/confident-ai/deepeval)
  579. - [ragas](https://github.com/explodinggradients/ragas)
  580. # Contribution
  581. We appreciate all the contributors for their efforts to improve and enhance `Dingo`. Please refer to the [Contribution Guide](docs/en/CONTRIBUTING.md) for guidance on contributing to the project.
  582. # License
  583. This project uses the [Apache 2.0 Open Source License](LICENSE).
  584. This project uses fasttext for some functionality including language detection. fasttext is licensed under the MIT License, which is compatible with our Apache 2.0 license and provides flexibility for various usage scenarios.
  585. # Citation
  586. If you find this project useful, please consider citing our tool:
  587. ```
  588. @misc{dingo,
  589. title={Dingo: A Comprehensive AI Data Quality Evaluation Tool for Large Models},
  590. author={Dingo Contributors},
  591. howpublished={\url{https://github.com/MigoXLab/dingo}},
  592. year={2024}
  593. }
  594. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.