Skip to content
Back to search

OpenVLHarness

Official Code for OpenVLHarness: Real-World Visual Agentic Tool‑Use with Multimodal Memory

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/uclanlp--openvlharness/badge.svg)](/listings/uclanlp--openvlharness)

HTML

<a href="/listings/uclanlp--openvlharness"><img src="/api/listings/uclanlp--openvlharness/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

Use Python **3.10 or newer**. From a source checkout:

git clone https://github.com/uclanlp/OpenVLHarness.git
cd OpenVLHarness
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[qwen,server]'

This installs the agent, Qwen tokenizer support, and the tool router. The model servers have separate environments and dependencies. For a client connecting to an existing router, `pip install -e .` is sufficient; add the `qwen` extra when using Qwen for local context counting or raw trace export. Add the `eval` extra to run benchmarks (see [Evaluation](#evaluation)).

<a id="quick-start"></a>

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

README.md · README · 715 lines

View this file on GitHub (read at the repository's current version: the scan did not record which commit it read)

This file has changed since it was scanned, so this is not exactly the text that was scanned, and no line is marked. The findings are listed with the safety scan above.

  1. <h1 align="center">
  2. <picture>
  3. <source media="(prefers-color-scheme: dark)" srcset="assets/logo-dark.svg">
  4. <img src="assets/logo.svg" alt="OpenVLHarness logo" height="44" align="top">
  5. </picture>
  6. OpenVLHarness
  7. </h1>
  8. <h3 align="center">Real World Visual Agentic Tool&#8209;Use with Multimodal Memory</h3>
  9. <p align="center">
  10. <a href="https://openvlharness.github.io/"><img src="https://img.shields.io/badge/Project-Homepage-2653c1.svg?style=for-the-badge" height="25" alt="Project homepage"></a>
  11. <a href="https://arxiv.org/abs/2610.26860"><img src="https://img.shields.io/badge/arXiv-2610.26860-b31b1b.svg?style=for-the-badge" height="25" alt="arXiv:2610.26860"></a>
  12. <a href="https://huggingface.co/papers/2610.26860"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Top%203-Paper-ffd21e.svg?style=for-the-badge" height="25" alt="Hugging Face Daily Papers: Top 3 Paper of the Day"></a>
  13. <a href="https://huggingface.co/spaces/uclanlp/OpenVLHarness"><img src="https://img.shields.io/badge/Interactive-Demo-8a2f6f.svg?style=for-the-badge" height="25" alt="Interactive demo"></a>
  14. <a href="https://huggingface.co/datasets/uclanlp/OpenVLHarness-Evaluation-Datasets"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Evaluation-Datasets-f1b33c.svg?style=for-the-badge" height="25" alt="Evaluation datasets on Hugging Face"></a>
  15. </p>
  16. <p align="center">
  17. <a href="https://openvlharness.github.io/">
  18. <picture>
  19. <source media="(prefers-color-scheme: dark)" srcset="assets/teaser-dark.png">
  20. <img src="assets/teaser.png" alt="Average score over 22 benchmarks for six backbones with and without OpenVLHarness, and GPT-6 Luna performance versus cost across five reasoning efforts" width="100%">
  21. </picture>
  22. </a>
  23. </p>
  24. <p align="center">
  25. <picture>
  26. <source media="(prefers-color-scheme: dark)" srcset="assets/tagline-dark.svg">
  27. <img src="assets/tagline.svg" alt="Teach any MLLM to think and code in visual space with precise multimodal evidence" width="660">
  28. </picture>
  29. </p>
  30. <a id="news"></a>
  31. ## 📰 News
  32. - **2026-11-13:** We will give a [talk on OpenVLHarness](https://mp.weixin.qq.com/s/Y0SOJ3esuFWU0GnR2p6q5w) on Friday, November 13, 2026, 6–7 PM PST. [Register here](https://t.bilibili.com/1268243648604012563).
  33. - **2026-10-24:** Listed on [Hugging Face Trending Papers](https://huggingface.co/papers/trending) and featured by [Hugging Papers](https://x.com/HuggingPapers/status/2114027064751685782) on X.
  34. - **2026-10-20:** [Featured by AK on X](https://x.com/_akhaliq/status/2112566784343876081).
  35. - **2026-10-20:** Top 3 Paper of the Day on [Hugging Face Daily Papers](https://huggingface.co/papers/date/2026-10-20).
  36. - **2026-10-19:** Paper released on [arXiv](https://arxiv.org/abs/2610.26860).
  37. <a id="highlights"></a>
  38. ## ✨ Highlights
  39. - **Consistent gains.** Across 22 benchmarks in four domains, seven backbones
  40. (GPT-6 Astra, Sol and Luna, GPT-5, Kimi K3, Qwen3-VL-32B and Qwen3-VL-8B) gain
  41. 9.7 to 12.6 points on average over the same models without tools.
  42. - **Cost efficiency.** With GPT-6 Luna, OpenVLHarness at low reasoning effort
  43. already averages 49.9 across the 22 benchmarks, ahead of the base model, Codex
  44. and Visual Sketchpad at max effort (48.0 to 48.8), while spending at most 26%
  45. of their model-token cost.
  46. - **Flexible capabilities.** Customizable capabilities and backend models based on
  47. your performance, cost, and latency preferences.
  48. [Choose the capabilities you want.](#configure-and-extend)
  49. - **Multimodal memory.** Images, boxes, masks, depth maps and web references stay
  50. addressable for the whole question, so later tools and code can build on them.
  51. - **Any model.** Gains validated on eight backbones across five open-weight and
  52. proprietary model families. Try any model of your choice by adding it to the
  53. [model registry](docs/models.md#other-registered-models).
  54. - **Training-free.** Works with the backbone's existing weights through native
  55. tool calling; nothing is fine-tuned.
  56. <a id="table-of-contents"></a>
  57. ## 🗂️ Table of Contents
  58. - [How it works](#how-it-works)
  59. - [Results](#results)
  60. - [Installation](#installation)
  61. - [Quick start](#quick-start)
  62. - [Evaluation](#evaluation)
  63. - [Configure and extend](#configure-and-extend)
  64. - [Repository layout](#repository-layout)
  65. - [Documentation and tests](#documentation-and-tests)
  66. - [Misc](#misc)
  67. - [Citation](#citation)
  68. <a id="how-it-works"></a>
  69. ## 🔍 How it works
  70. The model chooses tools, inspects their evidence, and continues until it can answer.
  71. Three components let it combine evidence across steps:
  72. - **Capability layer:** nine task-level interfaces turn tool outputs into readable
  73. observations, including annotated images, summaries, and compact numerical data.
  74. - **Multimodal memory:** images, structured data, and web references remain
  75. addressable throughout a question. Each new question starts with fresh memory.
  76. - **Python Coding Agent:** the backbone writes code in a separate context to
  77. calculate over stored boxes, masks, depth maps, and other tool results.
  78. For example, an agent can locate objects, estimate depth, and compute their spatial
  79. relationship; or crop an unfamiliar object, search for visual matches, and verify
  80. the answer on a webpage.
  81. | Capability | What it provides | Default backend |
  82. | --- | --- | --- |
  83. | Visual Grounding | Locate objects by text; return boxes, masks, and an annotated image | SAM 3 |
  84. | OCR | Read text in an image | PaddleOCR-VL |
  85. | Depth Estimation | Estimate per-pixel depth; retain the numerical array and visualization | Depth Anything 3 |
  86. | Camera Trajectory | Estimate camera poses across ordered images | Depth Anything 3 |
  87. | Zoom-in | Crop a region for closer inspection | Built-in image processing |
  88. | Text Search | Retrieve and summarize web, image, news, and other search results | Serper + Qwen |
  89. | Image Search | Find visual matches with reverse image search | Serper Lens + Qwen |
  90. | Webpage Visit | Extract evidence relevant to a goal from webpages | Serper scrape + Qwen |
  91. | Python Coding Agent | Generate and execute computations over stored tool results | Orchestrator backbone + local Python |
  92. Every preset uses **Qwen3-VL-8B-Instruct for multimodal search summarization and
  93. webpage extraction**, independently of the main model. The coding agent uses the
  94. same backbone as the main model.
  95. <a id="results"></a>
  96. ## 📈 Results
  97. Average score over the paper's 22 benchmarks (counting and grounding, multimodal
  98. search, general VQA and hallucination, and spatial understanding):
  99. | Backbone | Base | + OpenVLHarness | Gain |
  100. | --- | ---: | ---: | ---: |
  101. | Qwen3-VL-8B-Instruct | 34.24 | **45.41** | +11.17 |
  102. | Qwen3-VL-32B-Instruct | 38.30 | **50.92** | +12.62 |
  103. | Kimi K3 (max effort) | 56.43 | **66.09** | +9.66 |
  104. | GPT-5 (medium effort) | 44.37 | **55.65** | +11.28 |
  105. | GPT-6 Luna (medium effort) | 45.76 | **56.52** | +10.76 |
  106. | GPT-6 Sol (medium effort) | 52.67 | **64.31** | +11.64 |
  107. | GPT-6 Astra (medium effort) | 60.27 | **70.17** | +9.90 |
  108. Qwen3-VL results are the mean of three runs. Per-benchmark results, comparisons
  109. with other harnesses, and recorded trajectories are on the
  110. [project page](https://openvlharness.github.io/#results).
  111. <a id="installation"></a>
  112. ## 🛠️ Installation
  113. Use Python **3.10 or newer**. From a source checkout:
  114. ```bash
  115. git clone https://github.com/uclanlp/OpenVLHarness.git
  116. cd OpenVLHarness
  117. python -m venv .venv
  118. source .venv/bin/activate
  119. python -m pip install -e '.[qwen,server]'
  120. ```
  121. This installs the agent, Qwen tokenizer support, and the tool router. The model
  122. servers have separate environments and dependencies. For a client connecting to
  123. an existing router, `pip install -e .` is sufficient; add the `qwen` extra when
  124. using Qwen for local context counting or raw trace export. Add the `eval` extra to
  125. run benchmarks (see [Evaluation](#evaluation)).
  126. <a id="quick-start"></a>
  127. ## 🚀 Quick start
  128. ### 1. Pick a preset
  129. The release includes agent presets for seven of the paper's eight backbones:
  130. | Main model | Configuration | API |
  131. | --- | --- | --- |
  132. | Qwen3-VL-8B-Instruct | [qwen3_8b_paper.yaml](configs/agent/qwen3_8b_paper.yaml) | Chat Completions |
  133. | Qwen3-VL-32B-Instruct | [qwen3_32b_paper.yaml](configs/agent/qwen3_32b_paper.yaml) | Chat Completions |
  134. | GPT-5 | [gpt5_paper.yaml](configs/agent/gpt5_paper.yaml) | Responses |
  135. | GPT-6 Luna | [gpt6_luna_paper.yaml](configs/agent/gpt6_luna_paper.yaml) | Responses |
  136. | GPT-6 Sol | [gpt6_sol_paper.yaml](configs/agent/gpt6_sol_paper.yaml) | Responses |
  137. | GPT-6 Astra | [gpt6_astra_paper.yaml](configs/agent/gpt6_astra_paper.yaml) | Responses |
  138. | Kimi K3 | [kimi_k3_paper.yaml](configs/agent/kimi_k3_paper.yaml) | Chat Completions |
  139. The paper's Qwen2.5-VL-7B-Instruct results, compared with training-based harnesses
  140. in its appendix, used a separate configuration that this release does not include.
  141. The registry also covers every Qwen3-VL Instruct size (2B to 235B-A22B, with FP8
  142. builds) and OpenAI's GPT-5 to GPT-6.1 models; to use one, copy its family's
  143. preset and change one line ([other registered models](docs/models.md#other-registered-models)).
  144. ### 2. Prepare models and tools
  145. To run all nine capabilities, prepare these services before starting the agent:
  146. - **Models:** serve Qwen locally or connect to a GPT or Kimi endpoint, following
  147. the [model setup guide](docs/models.md). Every preset also needs a
  148. Qwen3-VL-8B-Instruct endpoint for search tools; the 8B preset can share it with
  149. the main model, while the 32B preset needs a second server for it. Sharing one
  150. server is the simplest deployment and scores within run-to-run noise of the
  151. paper setup; its cost is throughput, since search summarization and webpage
  152. extraction queue behind agent turns on the same GPU. To reproduce the paper
  153. exactly, or to keep agent throughput at higher concurrency, serve a second
  154. Qwen3-VL-8B instance and point `AUXILIARY_MODEL_BASE_URL` at it. The guide
  155. includes the tested model revision, serving versions, and GPU configuration.
  156. - **Perception:** launch SAM 3, PaddleOCR-VL, and Depth Anything 3 using the
  157. [tool service guide](openvlharness/services/tools/README.md). These run as
  158. separate GPU services behind the router.
  159. - **Search:** the three search tools call [Serper](https://serper.dev), a paid
  160. Google Search API with a free starter quota: web/image/news search, Google
  161. Lens reverse image search, and page scraping for webpage visits. Configure
  162. its key using `SERPER_API_KEY` or `SERPER_API_KEY_FILE`. Reverse image search
  163. additionally needs a URL Serper can fetch for the query image: either the
  164. built-in image host behind a tunnel (no cloud account) or an
  165. [Amazon S3](https://aws.amazon.com/s3/) or
  166. [Google Cloud Storage](https://cloud.google.com/storage) bucket, where the
  167. image is uploaded and handed to Serper as a signed URL. Follow the
  168. [search and storage setup](openvlharness/services/tools/serper_text_search/README.md).
  169. For the local Qwen setup, set these variables in both the router and agent
  170. terminals. Adjust the addresses to match your deployment:
  171. ```bash
  172. export MODEL_BASE_URL=http://127.0.0.1:8222/v1
  173. export MODEL_API_KEY=empty
  174. export AUXILIARY_MODEL_BASE_URL=http://127.0.0.1:8222/v1
  175. export AUXILIARY_MODEL_API_KEY=empty
  176. export JUDGE_MODEL=Qwen/Qwen3-VL-8B-Instruct
  177. export JUDGE_API_BASE=http://127.0.0.1:8222/v1/chat/completions
  178. export JUDGE_API_KEY=empty
  179. export TOOL_SERVER_URL=http://127.0.0.1:8101
  180. export SAM3_SERVER_URL=http://127.0.0.1:8086
  181. export OCR_SERVER_URL=http://127.0.0.1:8085
  182. export DEPTH_ESTIMATION_SERVER_URL=http://127.0.0.1:8087
  183. ```
  184. The `empty` keys above are for unauthenticated local endpoints. For GPT or Kimi,
  185. select its preset in the commands and Python example below, set `MODEL_BASE_URL`
  186. and `MODEL_API_KEY` for that provider, and keep the auxiliary connection pointing
  187. to Qwen. See the [connection examples](docs/models.md#connect-the-presets).
  188. ### 3. Run a question
  189. Start the router with the same configuration that the agent will use:
  190. ```bash
  191. python -m openvlharness.services.server --config configs/agent/qwen3_8b_paper.yaml
  192. ```
  193. In another terminal with the environment configured, check the router:
  194. ```bash
  195. curl --fail http://127.0.0.1:8101/health
  196. curl --fail http://127.0.0.1:8101/backend-status
  197. ```
  198. These checks only show that the router is up, how many tools it registered, and its
  199. concurrency limits. Unreachable GPU workers, missing Serper keys and missing image
  200. hosting settings surface on the first call to the affected tool. `/backend-status`
  201. also reports `degradations`: counts of search and webpage-visit calls that fell
  202. back to raw provider content because the auxiliary model timed out or returned
  203. unusable output, and image searches whose published query image could not be fetched
  204. at its public URL (`image_host`), and Serper calls rejected because of the account
  205. (`serper_account`: credits exhausted, key invalid or payment required). A healthy router
  206. keeps these at zero; the same events are logged with the tokens `SUMMARIZE_DEGRADED`,
  207. `EXTRACT_DEGRADED`, `IMAGE_HOST_UNREACHABLE` and `SERPER ACCOUNT-LEVEL FAILURE`.
  208. Then ask a question from Python:
  209. ```python
  210. from openvlharness.bootstrap import build_agent
  211. agent = build_agent("configs/agent/qwen3_8b_paper.yaml")
  212. result = agent.generate([
  213. {"type": "image", "value": "/path/to/image.jpg"},
  214. {"type": "text", "value": "How many bottles are visible?"},
  215. ])
  216. print(result["prediction"])
  217. trajectory = result["extra_records"]["trajectory"]
  218. ```
  219. Replace the image path with your own image. Inputs can include multiple images;
  220. image values can be local paths, HTTP(S) URLs, or data URLs. The returned trajectory
  221. records the model's turns, tool calls, and evidence for inspection.
  222. If `result["prediction"]` is `Error: Unable to get response from the server.`, the
  223. agent produced no answer. When a model request failed, the cause is in
  224. `result["extra_records"]["trajectory"]["error"]`; without that key, the model ended
  225. without a final answer. `Error: Reached the iteration limit without a final answer.`
  226. means the question used all `agent.max_iterations` iterations; the trajectory's `error`
  227. records the limit.
  228. > [!WARNING]
  229. > The coding tool runs generated Python in a subprocess on the agent host, with your
  230. > user's permissions and with network and filesystem access. Its execution timeout
  231. > is not a sandbox; run the agent in an isolated environment appropriate for your
  232. > data. See [SECURITY.md](SECURITY.md) and the
  233. > [coding tool README](openvlharness/services/tools/python_coding_agent/README.md).
  234. <a id="evaluation"></a>
  235. ## 📊 Evaluation
  236. ### Run a small evaluation
  237. Install the evaluation layer (benchmark loaders, judges and the runner) with the
  238. `eval` extra, keeping the model and tool services running:
  239. ```bash
  240. python -m pip install -e '.[eval,qwen]'
  241. ```
  242. This installs the dependencies of the `openvlharness-eval` command;
  243. `python run_benchmark.py` at the repository root is the same program. The
  244. evaluation code lives in [evaluation_kit](evaluation_kit) and needs no GPU packages.
  245. Use the same connection variables as above. `LMUData` is required: benchmark data
  246. downloads there, and the evaluator raises an error if it is unset. Put it and the
  247. work directory on a fast local disk. This example evaluates 12 reproducibly
  248. sampled questions **per dataset** from the FSC147 and ODinW validation splits,
  249. using direct counting and box metrics without an LLM judge:
  250. ```bash
  251. export LMUData=/path/to/local/datasets
  252. openvlharness-eval \
  253. --config configs/agent/qwen3_8b_paper.yaml \
  254. --judge-config configs/judge_no_judge.yaml \
  255. --data FSC147_val ODinW13_bbox_val_qwen3 \
  256. --num-examples 12 --shuffle --seed 42 \
  257. --num-proc 1 --save-html all --work-dir /path/to/local/runs
  258. ```
  259. `--num-proc` controls concurrent examples. Remove `--num-examples` and `--shuffle`
  260. to evaluate the full selected splits. For paper comparisons, also match the
  261. benchmark prompts, judge settings, and scoring protocol in Appendix D. The paper
  262. reports Qwen3-VL results averaged over three runs; a small subset checks the
  263. installation rather than reproducing those scores.
  264. ### Datasets and scoring
  265. Dataset names passed to `--data` select loaders in
  266. [evaluation_kit/dataset](evaluation_kit/dataset). The paper's 22 benchmarks
  267. (Table 1), the five additional search benchmarks of its appendix, and PixmoCount
  268. (supported but not reported in the paper) are:
  269. | Domain | Dataset names |
  270. | --- | --- |
  271. | Counting and grounding | `FSC147_val`, `CountQA`, `FSCD_147_bbox_val`, `FSCD_LVIS_bbox_val`, `ODinW13_bbox_val_qwen3` |
  272. | Multimodal search and deep research | `MMSearch`, `MMSearchPlus`, `RealXBench`, `FVQA_test`, `hle`, `BrowseCompVL` |
  273. | General VQA and hallucination | `RealWorldQA`, `SimpleVQA`, `HallusionBench`, `FREAK_VQA`, `FREAK_MCQ`, `VStarBench` |
  274. | Spatial understanding | `MMSIBench_wo_circular`, `ERQA`, `EmbSpatialBench`, `SparBench_tiny`, `MindCubeBench_tiny_raw_qa` |
  275. | Extended search (appendix) | `MMBrowseComp`, `VDRBench`, `VisBrowseBench`, `Infoseek_GDM_test`, `EncyclopedicVQA_test` |
  276. | Additional | `PixmoCount_val` |
  277. Every dataset above except `PixmoCount_val` and `CountQA` downloads, with its images,
  278. from [OpenVLHarness-Evaluation-Datasets](https://huggingface.co/datasets/uclanlp/OpenVLHarness-Evaluation-Datasets)
  279. into `LMUData` on first use; the loaders build those two from their original
  280. sources. `PixmoCount_val` is built from
  281. [allenai/pixmo-count](https://huggingface.co/datasets/allenai/pixmo-count), with
  282. images fetched from their original URLs; images that can no longer be fetched are
  283. left out, so a fresh build can contain fewer than the 534 images the paper evaluated.
  284. `CountQA` is built from
  285. [Jayant-Sravan/CountQA](https://huggingface.co/datasets/Jayant-Sravan/CountQA) on
  286. the Hugging Face Hub. ODinW also downloads evaluation annotations into `LMUData/GLIP`
  287. unless `GLIP_ROOT` is set. The
  288. [preparation scripts](evaluation_kit/dataset/scripts) document how the released
  289. splits were built.
  290. `FVQA_test`, `Infoseek_GDM_test` and `EncyclopedicVQA_test` are the largest search
  291. benchmarks (1,800, 2,000 and 1,960 questions). `FVQA_test_300`,
  292. `Infoseek_GDM_test_300` and `EncyclopedicVQA_test_300` are fixed 300-question subsets
  293. of them, a quick estimate of the full-split score at a fraction of the time and
  294. search cost; `SimpleVQA_300` is the 300 English SimpleVQA questions WebWatcher
  295. evaluates on.
  296. Benchmarks that need an LLM judge or answer extractor read it from `JUDGE_MODEL`,
  297. `JUDGE_API_BASE` (the full chat-completions URL) and `JUDGE_API_KEY` through
  298. [configs/judge.yaml](configs/judge.yaml); any OpenAI-compatible endpoint works. The
  299. paper judged with Qwen3-VL-8B-Instruct, and with GPT-4o on the search and
  300. deep-research benchmarks:
  301. ```bash
  302. export JUDGE_MODEL=gpt-4o
  303. export JUDGE_API_BASE=https://api.openai.com/v1/chat/completions
  304. export JUDGE_API_KEY=$OPENAI_API_KEY
  305. ```
  306. One judge serves every dataset of an invocation, so evaluate the search benchmarks in a
  307. separate `openvlharness-eval` run from the others, or re-score a finished run with
  308. `--mode eval --run-dir <run> --judge-config configs/judge.yaml` after switching the variables.
  309. <details>
  310. <summary>Grounding confidence scores</summary>
  311. [configs/judge_no_judge.yaml](configs/judge_no_judge.yaml) selects
  312. `score_mode: constant`, assigning every predicted box a confidence of 1.0.
  313. For ODinW, FSCD, and LVIS grounding, set `score_mode: original` to use matching
  314. tool detection scores. The configured value takes precedence over `SCORE_MODE`.
  315. With neither override, evaluators run both modes when detection sidecars exist
  316. (original is primary), or constant alone otherwise. The primary mode owns the
  317. unsuffixed metrics files; alternate metrics include the mode in their filenames.
  318. </details>
  319. ### Inspect and resume runs
  320. Results are saved under `<work-dir>/<model>/<agent>/<run_id>/<dataset>/`.
  321. For image/text benchmarks, the main artifacts are:
  322. | Artifact | Contents |
  323. | --- | --- |
  324. | `<model>.xlsx` | Predictions; set `PRED_FORMAT=tsv` or `PRED_FORMAT=json` to change the format |
  325. | Dataset-specific metric files | Scores and diagnostics written by the evaluator |
  326. | `full_trajectories/<sample_id>.json` | Agent messages, tool results and final answer |
  327. | `full_trajectories/images/` | Copied trajectory images; `--save-images base64` embeds images instead |
  328. | `tool_detections/<sample_id>.json` | Detection sidecars for samples with grounding results |
  329. | `htmls/<sample_id>.html` | Browsable tool evidence when `--save-html all` is supplied |
  330. | `<model>_incremental.jsonl` | Streaming results when `--incremental-json` is supplied |
  331. Prediction filenames use the model ID's final component, such as
  332. `Qwen3-VL-8B-Instruct`.
  333. The parent run directory contains `run_manifest.json`,
  334. `agent_config.redacted.yaml`, `judge_config.redacted.yaml`, and
  335. `tool_env_snapshot.json`, recording the effective configuration, resolved model
  336. profiles and service metadata. Add `--resume` to continue a compatible partial
  337. run; changes to the effective configuration require a new run. Rows that failed
  338. because a service was unreachable are scored as incorrect; once the service is
  339. back, rerun with `--run-dir <run-dir> --ignore-failed` to redo only those rows.
  340. To evaluate saved predictions, select the run directly:
  341. ```bash
  342. openvlharness-eval --mode eval --run-dir "$RUN_DIR" --data FSC147_val
  343. ```
  344. Evaluation reads the saved model identity, prediction format and judge configuration;
  345. it does not load the inference agent, tokenizer or tool connections. Supply
  346. `--judge-config` when changing the judge or replacing redacted judge credentials.
  347. Inference resumes still validate the effective configuration and resolved model
  348. profiles. Without `--run-dir`, evaluation discovers the latest complete saved run
  349. under `--work-dir`; use `--run-dir` when several model/agent identities are present.
  350. For **Qwen3-VL-8B-only token-level export**, add `--save-raw-trace`. This writes per-sample
  351. JSON and images under `raw_traces/`, with explicit error records for export
  352. failures. It requires a matching tokenizer and a server that returns token IDs;
  353. see [raw trace export](docs/models.md#raw-trace-export-qwen-only). Ordinary
  354. trajectory JSON and HTML reports work with every preset.
  355. Where files go:
  356. - Run outputs go under `--work-dir` (default `./outputs`).
  357. - Benchmark TSVs and images go in `$LMUData`.
  358. - Image archives and ODinW annotations are downloaded into the Hugging Face cache
  359. (`HF_HOME`, default `~/.cache/huggingface`) and then extracted into `$LMUData`,
  360. so they are stored twice: about 3.6 GB of archives for the datasets listed above.
  361. The CountQA and PixmoCount sources are cached there as well.
  362. - ODinW annotations go in `$LMUData/GLIP` unless `GLIP_ROOT` is set.
  363. - `openvlharness-eval` reads `./.env` if present, without overriding variables that
  364. are already set. The router and the Python API do not read it.
  365. - For inference, the default `--config` and `--judge-config` paths
  366. (`configs/agent/qwen3_8b_paper.yaml` and `configs/judge.yaml`) are relative, so
  367. run from the repository root or pass both.
  368. <a id="configure-and-extend"></a>
  369. ## 🧩 Configure and extend
  370. Each agent YAML separates `agent` policy, `memory`, `capabilities`, and `services`.
  371. Start from a copy of a paper preset and select the capabilities your deployment
  372. can support.
  373. ### Choose tools for your available infrastructure
  374. | Keys under `capabilities.tools` | Requirements in the paper setup |
  375. | --- | --- |
  376. | `grounding` | A SAM 3 GPU worker |
  377. | `ocr` | A PaddleOCR-VL GPU worker |
  378. | `depth`, `camera_pose` | A shared Depth Anything 3 GPU worker |
  379. | `text_search`, `web_visit` | A funded **Serper** API key and the auxiliary Qwen model endpoint |
  380. | `image_search` | Serper and auxiliary Qwen, plus a public URL for the query image: the built-in image host behind a tunnel, or an **Amazon S3 or Google Cloud Storage bucket** with upload/signing credentials |
  381. | `zoom` | CPU image processing in the router; no external API or GPU worker |
  382. | `coding` | The orchestrator model endpoint for code generation and local CPU Python execution |
  383. GPU workers can run on another machine reachable from the router. The main Qwen
  384. model and the auxiliary Qwen summarizer also need model serving, separately from
  385. these perception workers; a hosted main model still needs auxiliary Qwen when
  386. the search capabilities are enabled. See the [model](docs/models.md) and
  387. [tool deployment](openvlharness/services/tools/README.md) guides.
  388. Only reverse image search needs image hosting. Text search, including its
  389. image-search mode, and webpage visits do not. Configure `SERPER_API_KEY` or
  390. `SERPER_API_KEY_FILE` in the router environment, and follow the
  391. [S3/GCS credential instructions](openvlharness/services/tools/serper_text_search/README.md#credentials-and-reverse-image-upload)
  392. when enabling `image_search`.
  393. ### Disable capabilities you do not need
  394. 1. Remove the corresponding keys from `capabilities.tools`. For example, remove
  395. `image_search` to avoid the image-hosting requirement; remove `text_search`,
  396. `image_search`, and `web_visit` to disable Serper-backed tools altogether.
  397. Remove `grounding`, `ocr`, `depth`, and `camera_pose` to omit the GPU perception
  398. workers. Both depth and camera capabilities must be removed to stop needing DA3.
  399. 2. Delete the now-unused entries from `services.tools`. If all search capabilities
  400. are removed and nothing else uses it, also delete `services.models.auxiliary`.
  401. **Every `${VARIABLE}` remaining in the YAML is expanded during loading**, and
  402. every configured model is validated, even if its capability is disabled.
  403. Removing unused connections lets you leave their environment variables unset.
  404. 3. Restart the router with the edited file and pass that same file to
  405. `build_agent(...)` or `openvlharness-eval --config ...`. Check `/tools` and
  406. `/llm-schema` to confirm the selected tool set. Disabled capabilities are omitted
  407. from the model's tool schemas and their backend implementations are not loaded.
  408. <details>
  409. <summary>Example: zoom and coding with a hosted GPT model, without GPU tool workers, Serper, or cloud storage</summary>
  410. This configuration ships as [examples/local_tools.yaml](examples/local_tools.yaml).
  411. It needs only `MODEL_BASE_URL`, `MODEL_API_KEY`, and `TOOL_SERVER_URL` from the
  412. quick start; point the model connection to your GPT Responses endpoint. Code
  413. generation still calls that model endpoint.
  414. ```yaml
  415. agent_name: MainAgent
  416. agent_config_version: local_tools_v1
  417. agent:
  418. model_service: orchestrator
  419. max_iterations: 15
  420. memory:
  421. modules: [{name: image}, {name: tool_context}, {name: url}]
  422. emit_policy: always
  423. capabilities:
  424. registries: [paper]
  425. tools:
  426. zoom:
  427. backend: zoom_in_image_by_bbox
  428. coding:
  429. generator: python_coding_agent
  430. executor: python_executor_local
  431. services:
  432. client:
  433. base_url: ${TOOL_SERVER_URL}
  434. server:
  435. host: 127.0.0.1
  436. port: 8101
  437. registration: configured_only
  438. tools:
  439. zoom_in_image_by_bbox: {}
  440. python_coding_agent:
  441. model_service: orchestrator
  442. context: fresh
  443. python_executor_local:
  444. transport: local
  445. timeout_seconds: 30
  446. models:
  447. orchestrator:
  448. alias: gpt-6-luna
  449. base: ${MODEL_BASE_URL}
  450. key: ${MODEL_API_KEY}
  451. max_tokens: 16000
  452. reasoning_effort: medium
  453. ```
  454. Start it with
  455. `python -m openvlharness.services.server --config examples/local_tools.yaml`,
  456. then use `build_agent("examples/local_tools.yaml")`. Keep the configured
  457. router port aligned with `TOOL_SERVER_URL`.
  458. </details>
  459. ### Use another model
  460. Registered models are listed in the [model guide](docs/models.md#other-registered-models):
  461. copy the preset of the same family and change the orchestrator's `alias`. To add a
  462. model outside those families, register its served ID, API protocol, prompt, and
  463. supported options in [model_registry.yaml](openvlharness/services/model_registry.yaml).
  464. Auxiliary models can also use a registered compatible profile; multimodal Qwen
  465. remains the official search default.
  466. ### Connect your own service
  467. For SAM 3, OCR, DA3, or GroundingDINO workers, point
  468. `services.tools.<backend>.server_url` to a service implementing that backend's HTTP
  469. contract. For example, merge this fragment into your configuration to use a SAM 3
  470. worker on another host:
  471. ```yaml
  472. services:
  473. tools:
  474. sam3_segment:
  475. server_url: http://gpu-host:8086
  476. ```
  477. Use the service base URL; the backend wrapper adds the request route. The service
  478. must accept the wrapper's request format and return the expected result fields.
  479. The optional [GroundingDINO backend](openvlharness/services/tools/grounding_dino/README.md)
  480. shows how to select a different implementation of the same grounding capability.
  481. To integrate a service with a different API, add a wrapper under
  482. `openvlharness/services/tools/<backend>/tool.py`; the
  483. [OCR wrapper](openvlharness/services/tools/ocr/tool.py) is an HTTP example and
  484. the [zoom backend](openvlharness/services/tools/zoom_in_image_by_bbox/tool.py)
  485. is a small local example. Implement the backend's `BaseTool.execute` method,
  486. translate its response to the capability's expected fields, and register the
  487. backend as described below. Endpoints and credentials belong in deployment
  488. configuration or the service environment; public tool arguments describe the task.
  489. ### Add a capability or backend
  490. New capability names and backends require source registration before they can be
  491. selected in YAML. The relevant files are:
  492. 1. **Capability interface:** create `openvlharness/capabilities/<name>.py`,
  493. following [zoom.py](openvlharness/capabilities/zoom.py). Define the public
  494. function schema, a typed output contract, and a `Capability` subclass of
  495. [BaseCapability](openvlharness/capabilities/_base.py). Set `name`, `schema`,
  496. and `backends` (backend key to concrete tool class name). Implement
  497. `normalize_result` and `process_output` for canonical results and model-readable
  498. evidence. Use `translate_args`, `validate_result`, and `store_result` as needed
  499. to adapt arguments, validate results before registration, and retain numerical
  500. data for later tools. Image arguments such as `image_id` resolve through memory.
  501. 2. **Backend implementation:** add a package with `__init__.py` and `tool.py` under
  502. `openvlharness/services/tools/<backend>/`. Its tool class must match the
  503. capability's `backends` mapping. Implement
  504. [BaseTool](openvlharness/services/tools/base.py)'s `execute` method and return
  505. result data; the router adds the success/error envelope and tool-call ID.
  506. 3. **Configuration registration:** update `PAPER_CAPABILITIES`, `PAPER_BACKENDS`,
  507. and the backend/service-option validation in
  508. [bootstrap.py](openvlharness/bootstrap.py). For another implementation of
  509. an existing capability, extend its `backends` mapping and accepted backend
  510. selections. Add HTTP workers to `RAW_BACKENDS` when they need `server_url`
  511. injected from their service configuration.
  512. 4. **Router wiring:** [server.py](openvlharness/services/server.py) imports the
  513. configured backend class. Extend its construction logic if your tool needs
  514. injected model clients or other dependencies. For resource-limited services,
  515. assign a `BACKEND_GROUPS` entry and a `DEFAULT_LIMITS` concurrency limit. Keep
  516. capability and backend registration available on both the agent and router hosts.
  517. 5. **Select and verify:** add `capabilities.tools.<name>.backend` and the matching
  518. `services.tools.<backend>` settings to your YAML. Add focused coverage in
  519. [tests/test_capabilities.py](tests/test_capabilities.py) and
  520. [tests/test_services.py](tests/test_services.py) for valid calls, malformed
  521. arguments/results, and any stored image or numerical artifacts. Run a small
  522. question and inspect its trajectory to verify the model can use the evidence.
  523. See [runtime design](docs/runtime.md) for capability, memory, and service contracts.
  524. <a id="repository-layout"></a>
  525. ## 📁 Repository layout
  526. ```text
  527. openvlharness/ the harness (installed as the openvlharness package)
  528. agent/ orchestrator loop over model requests and tool calls
  529. capabilities/ the nine capability interfaces above
  530. memory/ per-question multimodal memory and Environment Update
  531. services/ model profiles, tool client and tool router
  532. tools/ tool backends (SAM 3, OCR, depth, search, Python executor, ...)
  533. prompts/ system prompts
  534. bootstrap.py builds an agent from a YAML config
  535. evaluation_kit/ benchmark loaders, judges and the runner (openvlharness-eval)
  536. configs/ paper agent presets and judge settings
  537. examples/ example agent configurations
  538. docs/ model, runtime and validation notes
  539. tests/ harness tests; tests/evaluation/ needs the eval extra
  540. validation/ fixed perception subset for checking a tool deployment
  541. assets/ logo and README figures
  542. run_benchmark.py same program as openvlharness-eval
  543. ```
  544. The harness never imports `evaluation_kit`; the evaluation layer uses the
  545. harness the same way any other application would.
  546. <a id="documentation-and-tests"></a>
  547. ## 📚 Documentation and tests
  548. | Guide | Contents |
  549. | --- | --- |
  550. | [Model services](docs/models.md) | Presets, registered models, Qwen serving, GPT and Kimi connections, per-role settings, raw trace export |
  551. | [Tool services](openvlharness/services/tools/README.md) | GPU workers, router deployment, search credentials, backend guides |
  552. | [Runtime design](docs/runtime.md) | Agent loop, configuration, memory, capability interfaces |
  553. | [Your model in the live demo](docs/demo_endpoint.md) | Serve a model with vLLM and get the Base URL and Model name for the demo's OpenAI-compatible option |
  554. | [Benchmarks and judges](evaluation_kit/dataset/README.md) | Benchmark loaders, metrics, judge configuration, `score_mode` |
  555. | [Release validation](docs/validation.md) | Regression coverage and live validation evidence |
  556. | [Security](SECURITY.md) | What runs where, what leaves your machine, reporting vulnerabilities |
  557. To run the package tests:
  558. ```bash
  559. python -m pip install -e '.[test,server]'
  560. python -m pytest tests -q
  561. ```
  562. The evaluator tests under `tests/evaluation` are collected only when the `eval`
  563. extra is installed as well.
  564. <a id="misc"></a>
  565. ## 📌 Misc
  566. <a id="star-history"></a>
  567. <p>
  568. <a href="https://www.star-history.com/#uclanlp/OpenVLHarness&Date">
  569. <img src="https://api.star-history.com/svg?repos=uclanlp/OpenVLHarness&type=Date" width="600" alt="Star history chart for uclanlp/OpenVLHarness">
  570. </a>
  571. </p>
  572. <a id="acknowledgements"></a>
  573. ### Acknowledgements
  574. The benchmark loaders and evaluators build on
  575. [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). The default tools use
  576. [SAM 3](https://github.com/facebookresearch/sam3),
  577. [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3),
  578. [PaddleOCR-VL](https://github.com/PaddlePaddle/PaddleOCR) and the
  579. [Serper](https://serper.dev/) search APIs. We thank their authors for releasing
  580. them.
  581. <a id="license"></a>
  582. ### License
  583. OpenVLHarness is released under the [MIT License](LICENSE). For third-party
  584. models and tools, see [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md).
  585. <a id="citation"></a>
  586. ## 📝 Citation
  587. If you use OpenVLHarness in your research, please cite:
  588. ```bibtex
  589. @article{openvlharness2026,
  590. title = {OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory},
  591. author = {Zhou, Yu and Yang, Cheng-Fu and Sun, Rui and Wu, Di and Peng, Sibo and Zhang, Mingyang and Yang, Zhen and Zhang, Ruiyi and Peng, Nanyun and Gan, Zhe and Chang, Kai-Wei},
  592. journal = {arXiv preprint arXiv:2610.26860},
  593. year = {2026},
  594. url = {https://arxiv.org/abs/2610.26860}
  595. }
  596. ```

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.