Skip to content
Back to search

lmms-eval

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today

Check it deeper on your machine

It has changed since its scan on 11 Nov 2026.

As a Claude Code plugin, install it and then ask Claude Code:

/plugin marketplace add keithmackay/mackayi/plugin install scanskillsafety@mackayiDo a deep scan of https://github.com/EvolvingLMMs-Lab/lmms-eval with scanskillsafety

Or from a terminal, clone it and run it:

git clone https://github.com/keithmackay/scanskillsafety-skill ~/.claude/skills/scanskillsafetynode ~/.claude/skills/scanskillsafety/cli.cjs --deep 'https://github.com/EvolvingLMMs-Lab/lmms-eval'

It checks the exact commit: every file, its packages against OSV (api.osv.dev), how it compares with this page's scan (findsafeskills.com receives the repository's address), and, if you agree, Claude reading every file with your own Claude subscription. It never runs the code.

Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/evolvinglmms-lab--lmms-eval/badge.svg)](/listings/evolvinglmms-lab--lmms-eval)

HTML

<a href="/listings/evolvinglmms-lab--lmms-eval"><img src="/api/listings/evolvinglmms-lab--lmms-eval/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Install and run your first evaluation in under 5 minutes:

git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
cd lmms-eval && uv pip install -e ".[all]"

# Run a quick evaluation (Qwen2.5-VL on MME, 8 samples)
python -m lmms_eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct \
  --tasks mme \
  --batch_size 1 \
  --limit 8

If it prints metrics, your environment is ready. For the full guide, see [`docs/getting-started/quickstart.md`](docs/getting-started/quickstart.md).

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 632 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <p align="center">
  2. <img src="https://raw.githubusercontent.com/EvolvingLMMs-Lab/lmms-eval/main/docs/images/lmms-eval-hero.avif" alt="LMMs-Eval" width="70%">
  3. </p>
  4. # LMMs-Eval: Probing Intelligence in the Real World
  5. [![PyPI](https://img.shields.io/pypi/v/lmms-eval)](https://pypi.org/project/lmms-eval)
  6. ![PyPI - Downloads](https://img.shields.io/pypi/dm/lmms-eval)
  7. [![GitHub contributors](https://img.shields.io/github/contributors/EvolvingLMMs-Lab/lmms-eval)](https://github.com/EvolvingLMMs-Lab/lmms-eval/graphs/contributors)
  8. [![issue resolution](https://img.shields.io/github/issues-closed-raw/EvolvingLMMs-Lab/lmms-eval)](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues)
  9. [![open issues](https://img.shields.io/github/issues-raw/EvolvingLMMs-Lab/lmms-eval)](https://github.com/EvolvingLMMs-Lab/lmms-eval/issues)
  10. > We are building the unified evaluation toolkit for frontier models and probing the abilities in real world, shape what we build next.
  11. <details>
  12. <summary>🌐 Available in 17 languages</summary>
  13. [简体中文](docs/i18n/README_zh-CN.md) | [繁體中文](docs/i18n/README_zh-TW.md) | [日本語](docs/i18n/README_ja.md) | [한국어](docs/i18n/README_ko.md) | [Español](docs/i18n/README_es.md) | [Français](docs/i18n/README_fr.md) | [Deutsch](docs/i18n/README_de.md) | [Português](docs/i18n/README_pt-BR.md) | [Русский](docs/i18n/README_ru.md) | [Italiano](docs/i18n/README_it.md) | [Nederlands](docs/i18n/README_nl.md) | [Polski](docs/i18n/README_pl.md) | [Türkçe](docs/i18n/README_tr.md) | [العربية](docs/i18n/README_ar.md) | [हिन्दी](docs/i18n/README_hi.md) | [Tiếng Việt](docs/i18n/README_vi.md) | [Indonesia](docs/i18n/README_id.md)
  14. </details>
  15. 📚 [Documentation](docs/README.md) | 📖 [100+ Tasks](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/docs/advanced/current_tasks.md) | 🌟 [30+ Models](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/models) | ⚡ [Quickstart](docs/getting-started/quickstart.md)
  16. 🏠 [Homepage](https://www.lmms-lab.com/) | 💬 [Discord](https://discord.gg/8xTM6jWnXa) | 🤝 [Contributing](CONTRIBUTING.md)
  17. ---
  18. ## Why `lmms-eval`?
  19. Benchmarks decide what gets built next. A model team that trusts its eval numbers can focus on real improvements instead of chasing noise. But the multimodal evaluation ecosystem is fragmented - scattered datasets, inconsistent post-processing, and single-number accuracy scores that hide whether a gain is real or random. Two teams evaluating the same model on the same benchmark routinely report different results.
  20. We believe [better evals lead to better models](https://arxiv.org/pdf/2211.09110). Good evaluation maps the border of what models can do and shapes what we build next.
  21. We are building `lmms-eval` and focusing on three core principles:
  22. - **Reproducible** - One pipeline, deterministic results. Same model, same benchmark, same numbers, every time.
  23. - **Efficient** - Evaluation should not be the bottleneck, even at large scale. Async serving, adaptive batching, and video I/O optimizations keep your GPUs saturated end to end.
  24. - **Trustworthy** - Not just accuracy. Confidence intervals, clustered standard errors, paired comparisons, and ongoing research into evaluation methodology. Results you can trust enough to act on.
  25. For how the pipeline works and the concrete mechanisms behind these principles, see [How the Evaluation Pipeline Works](docs/README.md#how-the-evaluation-pipeline-works) and [Why it's Efficient and Trustworthy](docs/README.md#why-its-efficient-and-trustworthy).
  26. ## What's New
  27. **v0.7** (Feb 2026) - Operational simplicity and pipeline maturity. 25+ new tasks across 8 domains, 2 new model backends, agentic task evaluation (`generate_until_agentic`), video I/O overhaul with TorchCodec (up to 3.58x faster), safety/red-teaming baselines, efficiency metrics (per-sample token counts, run-level throughput), and streamlined flattened JSONL log output for cleaner post-analysis. [Release notes](docs/releases/lmms-eval-0.7.md) | [Changelog](docs/releases/CHANGELOG.md).
  28. **v0.6** (Feb 2026) - Evaluation as a service. Standalone HTTP eval server, ~7.5x throughput over v0.5, statistically grounded results (CI, paired t-test), 50+ new tasks. [Release notes](docs/releases/lmms-eval-0.6.md) | [Changelog](docs/releases/CHANGELOG.md).
  29. **v0.5** (Oct 2025) - Audio expansion. Comprehensive audio evaluation, response caching, 50+ benchmark variants across audio, vision, and reasoning. [Release notes](docs/releases/lmms-eval-0.5.md).
  30. <details>
  31. <summary>Older updates</summary>
  32. - [2025-01] [Video-MMMU](https://arxiv.org/abs/2501.13826) - Knowledge acquisition from multi-discipline professional videos.
  33. - [2024-12] [MME-Survey](https://arxiv.org/pdf/2411.15296) - Comprehensive survey on evaluation of multimodal LLMs.
  34. - [2024-11] **v0.3** - Audio evaluation support (Qwen2-Audio, Gemini-Audio). [Release notes](docs/releases/lmms-eval-0.3.md).
  35. - [2024-06] **v0.2** - Video evaluation (LLaVA-NeXT Video, Gemini 1.5 Pro, VideoMME, EgoSchema). [Blog](https://lmms-lab.github.io/posts/lmms-eval-0.2/).
  36. - [2024-03] **v0.1** - First release. [Blog](https://lmms-lab.github.io/posts/lmms-eval-0.1/).
  37. </details>
  38. ## Quickstart
  39. Install and run your first evaluation in under 5 minutes:
  40. ```bash
  41. git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
  42. cd lmms-eval && uv pip install -e ".[all]"
  43. # Run a quick evaluation (Qwen2.5-VL on MME, 8 samples)
  44. python -m lmms_eval \
  45. --model qwen2_5_vl \
  46. --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct \
  47. --tasks mme \
  48. --batch_size 1 \
  49. --limit 8
  50. ```
  51. If it prints metrics, your environment is ready. For the full guide, see [`docs/getting-started/quickstart.md`](docs/getting-started/quickstart.md).
  52. ## Installation
  53. ### Using `uv` (Recommended for consistent environments)
  54. We use `uv` for package management to ensure all developers use exactly the same package versions. First, install uv:
  55. ```bash
  56. curl -LsSf https://astral.sh/uv/install.sh | sh
  57. ```
  58. For development with consistent environment:
  59. ```bash
  60. git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
  61. cd lmms-eval
  62. # Recommend
  63. uv pip install -e ".[all]"
  64. # If you want to use uv sync
  65. # uv sync # This creates/updates your environment from uv.lock
  66. ```
  67. To run commands:
  68. ```bash
  69. uv run python -m lmms_eval --help # Run any command with uv run
  70. ```
  71. To add new dependencies:
  72. ```bash
  73. uv add <package> # Updates both pyproject.toml and uv.lock
  74. ```
  75. ### Alternative Installation
  76. For direct usage from Git:
  77. ```bash
  78. uv venv eval
  79. uv venv --python 3.12
  80. source eval/bin/activate
  81. # You might need to add and include your own task yaml if using this installation
  82. uv pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
  83. ```
  84. <details>
  85. <summary>Reproduction of LLaVA-1.5's paper results</summary>
  86. You can check the [torch environment info](miscs/repr_torch_envs.txt) and [results check](miscs/llava_result_check.md) to **reproduce LLaVA-1.5's paper results**. We found torch/cuda versions difference would cause small variations in the results.
  87. </details>
  88. If you want to test on caption dataset such as `coco`, `refcoco`, and `nocaps`, you will need to have `java==1.8.0` to let pycocoeval api to work. If you don't have it, you can install by using conda
  89. ```
  90. conda install openjdk=8
  91. ```
  92. you can then check your java version by `java -version`
  93. <details>
  94. <summary>Comprehensive Evaluation Results of LLaVA Family Models</summary>
  95. <br>
  96. As demonstrated by the extensive table below, we aim to provide detailed information for readers to understand the datasets included in lmms-eval and some specific details about these datasets (we remain grateful for any corrections readers may have during our evaluation process).
  97. We provide a Google Sheet for the detailed results of the LLaVA series models on different datasets. You can access the sheet [here](https://docs.google.com/spreadsheets/d/1a5ImfdKATDI8T7Cwh6eH-bEsnQFzanFraFUgcS9KHWc/edit?usp=sharing). It's a live sheet, and we are updating it with new results.
  98. <p align="center" width="100%">
  99. <img src="https://i.postimg.cc/jdw497NS/WX20240307-162526-2x.png" width="100%" height="80%">
  100. </p>
  101. We also provide the raw data exported from Weights & Biases for the detailed results of the LLaVA series models on different datasets. You can access the raw data [here](https://docs.google.com/spreadsheets/d/1AvaEmuG4csSmXaHjgu4ei1KBMmNNW8wflOD_kkTDdv8/edit?usp=sharing).
  102. </details>
  103. <br>
  104. If you want to test [VILA](https://github.com/NVlabs/VILA), you should install the following dependencies:
  105. ```bash
  106. pip install s2wrapper@git+https://github.com/bfshi/scaling_on_scales
  107. ```
  108. Our Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub.
  109. ## Usage Examples
  110. > More examples can be found in [examples/models](examples/models)
  111. ### Evaluation with vLLM
  112. **Qwen2.5-VL:**
  113. ```bash
  114. bash examples/models/vllm_qwen2vl.sh
  115. ```
  116. **Qwen3-VL:**
  117. ```bash
  118. bash examples/models/vllm_qwen3vl.sh
  119. ```
  120. **Qwen3.5:**
  121. ```bash
  122. bash examples/models/vllm_qwen35.sh
  123. ```
  124. ### Evaluation with SGLang
  125. ```bash
  126. bash examples/models/sglang.sh
  127. ```
  128. **Qwen3.5:**
  129. ```bash
  130. bash examples/models/sglang_qwen35.sh
  131. ```
  132. **Wan2.2-I2V on VBVR:**
  133. ```bash
  134. bash examples/models/sglang_wan22_vbvr.sh
  135. ```
  136. ### Evaluation of OpenAI-Compatible Model
  137. ```bash
  138. bash examples/models/openai_compatible.sh
  139. ```
  140. ### Evaluation with Ollama (Local)
  141. ```bash
  142. bash examples/models/ollama.sh
  143. ```
  144. ### Evaluation with llama.cpp (Local)
  145. ```bash
  146. bash examples/models/llama_cpp.sh
  147. ```
  148. ### Evaluation of Qwen2.5-VL
  149. ```bash
  150. bash examples/models/qwen25vl.sh
  151. ```
  152. ### Evaluation of Qwen3-VL
  153. ```bash
  154. bash examples/models/qwen3vl.sh
  155. ```
  156. **More Parameters**
  157. ```bash
  158. python3 -m lmms_eval --help
  159. ```
  160. **Environmental Variables**
  161. Before running experiments and evaluations, we recommend you to export following environment variables to your environment. Some are necessary for certain tasks to run.
  162. ```bash
  163. export OPENAI_API_KEY="<YOUR_API_KEY>"
  164. export HF_HOME="<Path to HF cache>"
  165. export HF_TOKEN="<YOUR_API_KEY>"
  166. export HF_HUB_ENABLE_HF_TRANSFER="1"
  167. export REKA_API_KEY="<YOUR_API_KEY>"
  168. # Other possible environment variables include
  169. # ANTHROPIC_API_KEY,DASHSCOPE_API_KEY etc.
  170. ```
  171. **Common Environment Issues**
  172. Sometimes you might encounter some common issues for example error related to httpx or protobuf. To solve these issues, you can first try
  173. ```bash
  174. python3 -m pip install httpx==0.23.3;
  175. python3 -m pip install protobuf==3.20;
  176. # If you are using numpy==2.x, sometimes may cause errors
  177. python3 -m pip install numpy==1.26;
  178. # Sometimes sentencepiece is required for tokenizer to work
  179. python3 -m pip install sentencepiece;
  180. ```
  181. ## Custom Model Integration
  182. `lmms-eval` supports two types of models: **Chat (recommended)** and **Simple (legacy)**.
  183. ### Chat Models (Recommended) 🌟
  184. - Location: `lmms_eval/models/chat/`
  185. - Use: `doc_to_messages` function from task
  186. - Input: Structured `ChatMessages` with roles (`user`, `system`, `assistant`) and content types (`text`, `image`, `video`, `audio`)
  187. - Supports: Interleaved multimodal content
  188. - Uses: Model's `apply_chat_template()` method
  189. - Reference: `lmms_eval/models/chat/qwen2_5_vl.py` or `lmms_eval/models/chat/qwen3_vl.py`
  190. **Example input format:**
  191. ```python
  192. [
  193. {"role": "user", "content": [
  194. {"type": "image", "url": <image>},
  195. {"type": "text", "text": "What's in this image?"}
  196. ]}
  197. ]
  198. ```
  199. ### Simple Models (Legacy)
  200. - Location: `lmms_eval/models/simple/`
  201. - Use: `doc_to_visual` + `doc_to_text` functions from task
  202. - Input: Plain text with `<image>` placeholders + separate visual list
  203. - Supports: Limited (mainly images)
  204. - Manual processing: No chat template support
  205. - Reference: `lmms_eval/models/simple/instructblip.py`
  206. **Example input format:**
  207. ```python
  208. # Separate visual and text
  209. doc_to_visual -> [PIL.Image]
  210. doc_to_text -> "What's in this image?"
  211. ```
  212. ### Key Differences
  213. | Aspect | Chat Models | Simple Models |
  214. |--------|-------------|---------------|
  215. | File location | `models/chat/` | `models/simple/` |
  216. | Input method | `doc_to_messages` | `doc_to_visual` + `doc_to_text` |
  217. | Message format | Structured (roles + content types) | Plain text with placeholders |
  218. | Interleaved support | ✅ Yes | ❌ Limited |
  219. | Chat template | ✅ Built-in | ❌ Manual/None |
  220. | Recommendation | **Use this** | Legacy only |
  221. ### Why Use Chat Models?
  222. - ✅ Built-in chat template support
  223. - ✅ Interleaved multimodal content
  224. - ✅ Structured message protocol
  225. - ✅ Better video/audio support
  226. - ✅ Consistent with modern LLM APIs
  227. ### Chat Model Implementation Example
  228. ```python
  229. from lmms_eval.api.registry import register_model
  230. from lmms_eval.api.model import lmms
  231. from lmms_eval.protocol import ChatMessages
  232. @register_model("my_chat_model")
  233. class MyChatModel(lmms):
  234. is_simple = False # Use chat interface
  235. def generate_until(self, requests):
  236. for request in requests:
  237. # 5 elements for chat models
  238. doc_to_messages, gen_kwargs, doc_id, task, split = request.args
  239. # Get structured messages
  240. raw_messages = doc_to_messages(self.task_dict[task][split][doc_id])
  241. messages = ChatMessages(messages=raw_messages)
  242. # Extract media and apply chat template
  243. images, videos, audios = messages.extract_media()
  244. hf_messages = messages.to_hf_messages()
  245. text = self.processor.apply_chat_template(hf_messages)
  246. # Generate...
  247. ```
  248. For more details, see the [Model Guide](docs/guides/model_guide.md).
  249. ## Custom Dataset Integration
  250. ### Task Configuration with `doc_to_messages`
  251. Implement `doc_to_messages` to transform dataset documents into structured chat messages:
  252. ```python
  253. def my_doc_to_messages(doc, lmms_eval_specific_kwargs=None):
  254. # Extract visuals and text from doc
  255. visuals = my_doc_to_visual(doc)
  256. text = my_doc_to_text(doc, lmms_eval_specific_kwargs)
  257. # Build structured messages
  258. messages = [{"role": "user", "content": []}]
  259. # Add visuals first
  260. for visual in visuals:
  261. messages[0]["content"].append({"type": "image", "url": visual})
  262. # Add text
  263. messages[0]["content"].append({"type": "text", "text": text})
  264. return messages
  265. ```
  266. ### YAML Configuration
  267. ```yaml
  268. task: "my_benchmark"
  269. dataset_path: "my-org/my-dataset"
  270. test_split: test
  271. output_type: generate_until
  272. # For chat models (recommended)
  273. doc_to_messages: !function utils.my_doc_to_messages
  274. # OR legacy approach:
  275. doc_to_visual: !function utils.my_doc_to_visual
  276. doc_to_text: !function utils.my_doc_to_text
  277. process_results: !function utils.my_process_results
  278. metric_list:
  279. - metric: acc
  280. ```
  281. ### Key Features
  282. #### `doc_to_messages`
  283. - Transforms dataset document into structured chat messages
  284. - Returns: List of message dicts with `role` and `content`
  285. - Content supports: `text`, `image`, `video`, `audio` types
  286. - Protocol: Defined in `lmms_eval/protocol.py` (`ChatMessages` class)
  287. - Auto-fallback: If not provided, uses `doc_to_visual` + `doc_to_text`
  288. For more details, see the [Task Guide](docs/guides/task_guide.md).
  289. ## Web UI
  290. LMMS-Eval includes an optional Web UI for interactive evaluation configuration.
  291. ### Requirements
  292. - Node.js 18+ (for building the frontend, auto-built on first run)
  293. ### Usage
  294. ```bash
  295. # Start the Web UI (opens browser automatically)
  296. uv run lmms-eval-ui
  297. # Custom port
  298. LMMS_SERVER_PORT=3000 LMMS_EVAL_TUI_ALLOWED_ORIGINS=http://localhost:3000 uv run lmms-eval-ui
  299. ```
  300. The web UI provides:
  301. - Model selection from all available models
  302. - Task selection with search/filter
  303. - Real-time command preview
  304. - Live evaluation output streaming
  305. - Start/Stop evaluation controls
  306. - Log Viewer for browsing saved evaluation results and samples
  307. For more details, see [Web UI README](lmms_eval/tui/README.md).
  308. ## HTTP Evaluation Server
  309. LMMS-Eval includes a production-ready HTTP server for remote evaluation workflows.
  310. ### Why Use Eval Server?
  311. - **Decoupled evaluation**: Run evaluations on dedicated GPU nodes while training continues
  312. - **Async workflow**: Submit jobs without blocking training loops
  313. - **Queue management**: Sequential job processing with automatic resource management
  314. - **Remote access**: Evaluate models from any machine
  315. ### Start Server
  316. ```python
  317. from lmms_eval.entrypoints import ServerArgs, launch_server
  318. # Configure server
  319. args = ServerArgs(
  320. host="0.0.0.0",
  321. port=8000,
  322. max_completed_jobs=200,
  323. temp_dir_prefix="lmms_eval_"
  324. )
  325. # Launch server
  326. launch_server(args)
  327. ```
  328. Server runs at `http://host:port` with auto-generated API docs at `/docs`
  329. ### Client Usage
  330. **Sync Client:**
  331. ```python
  332. from lmms_eval.entrypoints import EvalClient
  333. client = EvalClient("http://eval-server:8000")
  334. # Submit evaluation (non-blocking)
  335. job = client.evaluate(
  336. model="qwen2_5_vl",
  337. tasks=["mmmu_val", "mme"],
  338. model_args={"pretrained": "Qwen/Qwen2.5-VL-7B-Instruct"},
  339. num_fewshot=0,
  340. batch_size=1,
  341. device="cuda:0",
  342. )
  343. # Continue training...
  344. # Later, retrieve results
  345. result = client.wait_for_job(job["job_id"])
  346. print(result["result"])
  347. ```
  348. **Async Client:**
  349. ```python
  350. from lmms_eval.entrypoints import AsyncEvalClient
  351. async with AsyncEvalClient("http://eval-server:8000") as client:
  352. job = await client.evaluate(
  353. model="qwen3_vl",
  354. tasks=["mmmu_val"],
  355. model_args={"pretrained": "Qwen/Qwen3-VL-4B-Instruct"},
  356. )
  357. result = await client.wait_for_job(job["job_id"])
  358. ```
  359. ### Server API Endpoints
  360. | Endpoint | Method | Description |
  361. |----------|--------|-------------|
  362. | `/health` | GET | Server health check |
  363. | `/evaluate` | POST | Submit evaluation job |
  364. | `/jobs/{job_id}` | GET | Get job status and results |
  365. | `/queue` | GET | View queue status |
  366. | `/tasks` | GET | List available tasks |
  367. | `/models` | GET | List available models |
  368. | `/jobs/{job_id}` | DELETE | Cancel queued job |
  369. | `/merge` | POST | Merge FSDP2 sharded checkpoints |
  370. ### Example Workflow
  371. ```python
  372. # Training loop pseudocode
  373. for epoch in range(num_epochs):
  374. train_one_epoch()
  375. # After every N epochs, evaluate checkpoint
  376. if epoch % 5 == 0:
  377. checkpoint_path = f"checkpoints/epoch_{epoch}"
  378. # Submit async evaluation (non-blocking)
  379. eval_job = client.evaluate(
  380. model="vllm",
  381. model_args={"model": checkpoint_path},
  382. tasks=["mmmu_val", "mathvista"],
  383. )
  384. # Training continues immediately
  385. print(f"Evaluation job submitted: {eval_job['job_id']}")
  386. # After training completes, retrieve all results
  387. results = []
  388. for job_id in eval_jobs:
  389. result = client.wait_for_job(job_id)
  390. results.append(result)
  391. ```
  392. ### Security Note
  393. ⚠️ **This server is intended for trusted environments only**. Do NOT expose to untrusted networks without additional security layers (authentication, rate limiting, network isolation).
  394. For more details, see the [v0.6 release notes](docs/releases/lmms-eval-0.6.md).
  395. ## Frequently Asked Questions
  396. <details>
  397. <summary><strong>What models does lmms-eval support?</strong></summary>
  398. We support 30+ model families out of the box, including Qwen2.5-VL, Qwen3-VL, LLaVA-OneVision, InternVL-2, VILA, and more. Any OpenAI-compatible API endpoint is also supported. See the full list in [`lmms_eval/models/`](lmms_eval/models/).
  399. Qwen3.5 is supported through existing runtime backends (`--model vllm` and `--model sglang`) by setting `model=Qwen/Qwen3.5-397B-A17B` in `--model_args`.
  400. The Qwen3.5 example scripts align with official runtime references (for example, `max_model_len/context_length=262144` and `reasoning_parser=qwen3`).
  401. If a new model family is already fully supported by vLLM or SGLang at runtime, we generally only need documentation and examples instead of adding a dedicated model wrapper.
  402. </details>
  403. <details>
  404. <summary><strong>What benchmarks and tasks are available?</strong></summary>
  405. Over 100 evaluation tasks across image, video, and audio modalities, including MMMU, MME, MMBench, MathVista, VideoMME, EgoSchema, and many more. Check [`docs/advanced/current_tasks.md`](docs/advanced/current_tasks.md) for the full list.
  406. </details>
  407. <details>
  408. <summary><strong>How do I add my own benchmark?</strong></summary>
  409. Create a YAML config under `lmms_eval/tasks/` with dataset path, splits, and a `doc_to_messages` function. See [`docs/guides/task_guide.md`](docs/guides/task_guide.md) for a step-by-step guide.
  410. </details>
  411. <details>
  412. <summary><strong>Can I evaluate a model behind an API (e.g., GPT-4o, Claude)?</strong></summary>
  413. Yes. Use `--model openai` with `--model_args model=gpt-4o` and set `OPENAI_API_KEY`. Any OpenAI-compatible endpoint works, including local vLLM/SGLang/Ollama/llama.cpp servers.
  414. </details>
  415. <details>
  416. <summary><strong>How do I run evaluations on multiple GPUs?</strong></summary>
  417. Use `accelerate launch` or pass `--device cuda` with tensor parallelism via vLLM/SGLang backends. See [`docs/getting-started/commands.md`](docs/getting-started/commands.md) for multi-GPU flags.
  418. </details>
  419. <details>
  420. <summary><strong>How do I cite lmms-eval?</strong></summary>
  421. Use the BibTeX entries below, or click the "Cite this repository" button in the GitHub sidebar (powered by our [`CITATION.cff`](CITATION.cff)).
  422. </details>
  423. ## Acknowledgement
  424. lmms_eval is a fork of [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness). We recommend you to read through the [docs of lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs) for relevant information.
  425. ---
  426. Below are the changes we made to the original API:
  427. - Build context now only pass in idx and process image and doc during the model responding phase. This is due to the fact that dataset now contains lots of images and we can't store them in the doc like the original lm-eval-harness otherwise the cpu memory would explode.
  428. - Instance.args (lmms_eval/api/instance.py) now contains a list of images to be inputted to lmms.
  429. - lm-eval-harness supports all HF language models as single model class. Currently this is not possible for lmms because the input/output format of lmms in HF are not yet unified. Therefore, we have to create a new class for each lmms model. This is not ideal and we will try to unify them in the future.
  430. ---
  431. ## Citations
  432. ```bibtex
  433. @misc{zhang2024lmmsevalrealitycheckevaluation,
  434. title={LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models},
  435. author={Kaichen Zhang and Bo Li and Peiyuan Zhang and Fanyi Pu and Joshua Adrian Cahyono and Kairui Hu and Shuai Liu and Yuanhan Zhang and Jingkang Yang and Chunyuan Li and Ziwei Liu},
  436. year={2024},
  437. eprint={2407.12772},
  438. archivePrefix={arXiv},
  439. primaryClass={cs.CL},
  440. url={https://arxiv.org/abs/2407.12772},
  441. }
  442. @misc{lmms_eval2024,
  443. title={LMMs-Eval: Accelerating the Development of Large Multimodal Models},
  444. url={https://github.com/EvolvingLMMs-Lab/lmms-eval},
  445. author={Bo Li*, Peiyuan Zhang*, Kaichen Zhang*, Fanyi Pu*, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li and Ziwei Liu},
  446. publisher = {Zenodo},
  447. version = {v0.1.0},
  448. month={March},
  449. year={2024}
  450. }
  451. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.