Skip to content
Back to search

https://github.com/AppThreat/vulnerability-db

SkillWorks with Claude CodeGitHubIncomplete

Safety scan

A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.

Checked today
Badge for your README
findsafeskills safety rating

Markdown

[![findsafeskills safety rating](/api/listings/appthreat--vulnerability-db/badge.svg)](/listings/appthreat--vulnerability-db)

HTML

<a href="/listings/appthreat--vulnerability-db"><img src="/api/listings/appthreat--vulnerability-db/badge.svg" alt="findsafeskills safety rating" /></a>

Install

i

Install instructions from the README

As written by the repo's author. findsafeskills has not verified these steps.

pip install appthreat-vulnerability-db>=7.0.0

For optional dependencies such as `oras`, install the `[oras]` or `[all]` group:

pip install appthreat-vulnerability-db[all]

Users of dep-scan v5 need the 5.8.x line instead:

pip install appthreat-vulnerability-db==5.8.0

What was scanned

The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.

SKILL.md · Manifest · 348 lines
  1. # SKILL.md
  2. A practical guide for AI agents that need to use `vdb`. If you are changing the
  3. code rather than using it, read [AGENTS.md](./AGENTS.md) instead.
  4. Coming from VDB 6.x, read [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) before
  5. acting on any 6.x instructions you find elsewhere. Image URLs, environment
  6. variables and the comparison API changed, v7 refuses to read v6 databases, and
  7. 6.x code samples contain one API call that no longer exists.
  8. ## The rule that matters most
  9. An empty result is not a clean result until you have checked that the database
  10. exists, covers the purl types you asked about, and is fresh enough to matter.
  11. VDB reports all three, and the reports exist because a confident empty answer
  12. is the most dangerous output a vulnerability scanner can produce.
  13. Concretely, before you tell a user "no vulnerabilities found":
  14. ```mermaid
  15. flowchart TD
  16. A[Search returned no findings] --> B{Exit code 0?}
  17. B -- "no, exit 1 + 'empty'" --> X[Scanner down.<br/>Report a failure, not a clean scan.]
  18. B -- yes --> C{Shard store?}
  19. C -- "no, full database" --> D{Metadata search?}
  20. C -- yes --> E{Every requested purl<br/>type covered?}
  21. E -- "no, coverage_gap" --> Y[NOT CHECKED.<br/>Name the uncovered types.]
  22. E -- yes --> F{Any answering<br/>shard stale?}
  23. F -- yes --> Z[Answer may be old.<br/>Name the stale shard.]
  24. F -- no --> D
  25. D -- "yes, on a database<br/>without metadata" --> W[Tells you nothing.<br/>Build with --include-metadata.]
  26. D -- no --> OK[Clean result.<br/>Safe to report.]
  27. ```
  28. Only the `OK` branch is a clean result. Every other leaf is a statement about
  29. the database rather than about the software being scanned.
  30. ## Skill: choose the right database
  31. v7 publishes `vdb7-full`, `vdb7-app-only`, five app-scope variants, and
  32. per-type shards.
  33. | Need | Artifact |
  34. | :------------------------------------------------ | :------------------------------------------------------- |
  35. | Standard application dependency scanning | `vdb7-app-only` |
  36. | Container or Linux OS package scanning | `vdb7-full` |
  37. | CVE ID, alias, CPE, reference or full-text search | `vdb7-full` only; these raise on a shard |
  38. | Single-ecosystem workload (npm only, maven only) | the matching type shard |
  39. | Metadata, text, alias, reference or symbol search | `vdb7-app-extended` or `vdb7-app-10y-extended` |
  40. | Advisories older than 2020 | `vdb7-app-10y` / `vdb7-app-10y-extended` (NVD from 2016) |
  41. | Smallest download, recent advisories (CI) | `vdb7-app-2y` / `vdb7-app-2y-extended` (NVD from 2024) |
  42. `vdb7-app-only` is the app-scope _database_. `vdb7-app` is a different
  43. artifact: the app group _shard_, `completeness=partial`, for the shard store
  44. only. `vdb db refresh full` rejects it by design.
  45. Five shards are structural (`deb`, `rpm`, `apk`, `app`, and `cpe` as the
  46. complement) and the rest are per-type. There are 43 possible shard names and a
  47. build emits only the ones it has rows for, which was 26 on a recent app+OS
  48. build. Do not assume a fixed shard list; a shard's `vdb.meta`
  49. (`artifact.types`, `siblings.available`) is the authority, and `vdb db status`
  50. reports what is actually local.
  51. Prefer an organization-controlled mirror or an internally produced artifact
  52. when the user has one. Use the AppThreat-hosted defaults for bootstrap, local
  53. testing, or when no internal URL was given.
  54. Default v7 databases keep `cve_metadata` and `cve_metadata_text` empty to
  55. hold the index size down; the extended flavors (`vdb7-app-extended`,
  56. `vdb7-app-2y-extended`, `vdb7-app-10y-extended`) are the pre-built exceptions. When a task needs
  57. `search_full_text()`, `search_by_alias()`, `search_by_reference()`,
  58. `search_by_package_name()`, `search_by_symbol()`, or a metadata-dependent
  59. filter (source, severity, dates, malware flags), download the extended flavor
  60. with `vdb db refresh full --flavor app-extended`, build locally with
  61. `--include-metadata`, or point `VDB_DATABASE_URL` at an internal metadata
  62. artifact. On a default database those searches fail safe by returning no
  63. metadata matches, so an empty result there tells you nothing about the package.
  64. ## Skill: download before searching
  65. The files are `data.vdb7` and `data.index.vdb7` under `VDB_HOME`. v7 does not
  66. read or migrate `.vdb6`, and a search against a zero-row database exits 1 with
  67. an empty-database message rather than printing "No results found!".
  68. ```bash
  69. vdb db refresh full --app-only # app-only
  70. vdb db refresh full # app+OS
  71. vdb db refresh full --flavor app-extended # app-only + metadata tables
  72. ```
  73. Both stage and validate the download before it replaces the local database.
  74. For type shards:
  75. ```bash
  76. vdb db refresh npm pypi # named shards into the store
  77. ```
  78. The `-xz` image suffix is the safe choice: it always unpacks. A `-zst`
  79. reference works only when a zstd decompressor is present (Python 3.14+, or a
  80. `zstd` binary on PATH); without one the refresh fails before downloading and
  81. names both remedies. `vdb db refresh --compression zst` selects zst
  82. explicitly.
  83. In Python:
  84. ```python
  85. import os
  86. from vdb.lib import config, db, search
  87. from vdb.lib.orasclient import download_image
  88. DB_URL = os.getenv("VDB_APP_ONLY_DATABASE_URL", config.VDB_APP_ONLY_DATABASE_URL)
  89. if db.needs_update(days=1, default_status=True):
  90. download_image(DB_URL, config.DATA_DIR)
  91. results = search.search_by_any("pkg:pypi/requests@2.31.0", with_data=True)
  92. ```
  93. Set `VDB_HOME` before importing `vdb.lib.config`. Path and build-scope
  94. variables are read at import time.
  95. ## Skill: get structured results
  96. The CLI prints human-readable tables. There is no `--json` flag on the search
  97. commands, so do not look for one. Use the Python API for machine-readable
  98. output.
  99. ```python
  100. import json
  101. from vdb.lib import search
  102. output = []
  103. for item in search.search_by_any("pkg:npm/lodash@4.17.20", with_data=True):
  104. row = {
  105. "cve_id": item["cve_id"],
  106. "package": item["name"],
  107. "purl_prefix": item["purl_prefix"],
  108. "vers": item["vers"],
  109. "fix_version": item.get("fix_version"),
  110. "severity": item.get("severity"),
  111. }
  112. if item.get("source_data"):
  113. row["cve_data"] = item["source_data"]
  114. output.append(row)
  115. print(json.dumps(output, indent=2))
  116. ```
  117. `source_data` is a plain dict, ready for `json.dumps()`. It is not a Pydantic
  118. model and has no `.model_dump()`. Older snippets call
  119. `item["source_data"].model_dump(mode="json")`; that raises `AttributeError` on
  120. v7. This is the single most common wrong line in 6.x-era sample code.
  121. ## Skill: bulk searches
  122. ```python
  123. from vdb.lib import search
  124. packages = [
  125. {"purl": "pkg:pypi/requests@2.31.0"},
  126. {"purl": "pkg:npm/react@18.2.0"},
  127. {"url": "https://github.com/pallets/flask"},
  128. ]
  129. for batch in search.search_packages_batched(packages, batch_size=100, with_data=False):
  130. for item in batch:
  131. print(item["locator"], item["result_count"], item.get("max_severity"))
  132. ```
  133. For a CycloneDX BOM, `search_bom_summary` and `search_bom_detailed` take the
  134. same filters, and `search_by_cdx_bom` yields batches so peak memory stays
  135. bounded on a large SBOM.
  136. ## Skill: manage local data with `vdb db`
  137. ```bash
  138. vdb db status # main DB and per-shard build id, age, size, coverage
  139. vdb db status --json # the same report, machine-readable
  140. vdb db refresh # sync what I have: everything already in the store
  141. vdb db refresh npm pypi # a named set of shards (names or purl types)
  142. vdb db refresh --all # every default shard
  143. vdb db refresh full # the full image, staged and validated
  144. ```
  145. Call `vdb db status` before reporting scan results, and again after any refresh.
  146. The plain and JSON reports expose the same facts: per-shard `build_id`, build
  147. time and `age_days`, sizes, `mixed_builds`, `missing_group_shards`, `coverage`
  148. (`full`, or `partial (missing: ...)`), and a per-shard `stale` flag.
  149. Refreshes are safe to interrupt. The full image is downloaded to a staging
  150. directory, validated (both `.vdb7` files present, index non-empty, not a
  151. single-shard artifact) and swapped in only after validation, with the current
  152. files stashed away first. Each shard lands with an atomic rename. A failed or
  153. empty fetch is an error and never replaces a working database.
  154. ## Skill: read coverage and staleness honestly
  155. When the connected main database is a shard (`vdb.meta` says
  156. `completeness: "partial"`), purl searches fan out across the local shard store.
  157. Two conditions that look similar and are not:
  158. **Coverage gap.** A requested purl type has no local shard. Single-purl
  159. searches raise `ShardCoverageError`. Batch searches mark the affected
  160. components `coverage_gap: true`, expose requested, covered and uncovered types
  161. through the `.coverage` companion on the result list and
  162. `vdb.lib.search.last_shard_coverage`, and warn on stderr.
  163. **Staleness.** A shard more than 7 days older than its newest sibling. Reported
  164. per shard by `vdb db status` and warned at search time. A shard with no
  165. parseable build time is never claimed stale, though its `build_id` is still
  166. shown.
  167. Rules for an agent:
  168. An empty result on a shard store is not "no vulnerabilities" until `coverage`
  169. shows every requested type was covered. A `coverage_gap` means not checked.
  170. A stale shard still answers, and the answer may be old. Name the stale shard
  171. when you report results.
  172. Never conclude "clean" from a scan whose `vdb db status` shows
  173. `missing_group_shards`.
  174. Never enable `VDB_AUTO_FETCH` on a user's behalf in air-gapped CI. On-demand
  175. network fetch during a scan is opt-in for a reason.
  176. ## Skill: build a local database safely
  177. Use app-only builds unless OS package scanning is required.
  178. ```bash
  179. export VDB_HOME=/tmp/vdb-home
  180. export VDB_CACHE=/tmp/vdb-cache
  181. export VDB_TEMP_DIR=/tmp/vdb-home/tmp
  182. export NVD_START_YEAR=2024
  183. export GITHUB_PAGE_COUNT=1
  184. mkdir -p "$VDB_HOME" "$VDB_CACHE" "$VDB_TEMP_DIR"
  185. vdb --cache
  186. ```
  187. `NVD_START_YEAR` bounds NVD-style CVE data and the Linux distro feeds only.
  188. Application ecosystem advisories are not year-filtered, because a distro fix
  189. moves the whole release forward while a lockfile does not, so an old library
  190. advisory is exactly what a scan needs. Set
  191. `VDB_APP_ECOSYSTEM_START_YEAR` if you specifically want the smaller artifact.
  192. Do not set `OSV_EXCLUDE_MALWARE=true` to save space on a build you will scan
  193. with. It removes the malicious-package advisories, and a missed malicious
  194. package is an installed backdoor rather than a stale finding.
  195. For app+OS builds, and for metadata search support:
  196. ```bash
  197. vdb --cache-os
  198. vdb --cache --include-metadata
  199. vdb --cache-os --include-metadata
  200. ```
  201. Builds print minimal progress. Use `--quiet` or `VDB_QUIET=true` to suppress
  202. the logo, logs and progress. Tune cadence with `VDB_PROGRESS_INTERVAL`.
  203. To restrict OS data, use the `VDB_IGNORE_*` and `VDB_INCLUDE_*` flags from
  204. `vdb/lib/config.py`:
  205. ```bash
  206. export VDB_IGNORE_DEBIAN=true
  207. export VDB_IGNORE_ALPINE=true
  208. vdb --cache-os
  209. ```
  210. Set `VDB_TEMP_DIR` to a partition with room. An app+OS build needs significant
  211. temporary space for index creation and `VACUUM`, and a small `/tmp` is the most
  212. common build failure.
  213. ## Skill: reproduce cache bugs without large downloads
  214. `AquaSource` reads `$VDB_CACHE/vuln-list.zip` when it exists. Build a small zip
  215. with a few representative paths and monkeypatch `config.CACHE_DIR` at it in
  216. tests. This avoids cloning or downloading the full vuln-list repository.
  217. ## Skill: reason about v7 storage
  218. `data.vdb7` stores each CVE source blob once in `cve_source_data`, with package
  219. locator rows in `cve_data` referencing them by hash. `data.index.vdb7` stores
  220. package and version rows in `cve_index`, plus optional metadata in
  221. `cve_metadata` (compact, package-level) and `cve_metadata_text` (long text, one
  222. row per CVE).
  223. The split follows access patterns: the index is small, random-access and hot;
  224. the data file is large, sequential and cold. Resolve against the index first,
  225. hydrate only what matched.
  226. Databases are built from scratch by the release workflows. Schema migrations
  227. and legacy inline `cve_data.source_data` write paths are deliberately not
  228. required. v7 reads `.vdb7` only.
  229. ## Skill: direct SQLite inspection
  230. The `.vdb7` files are ordinary SQLite databases.
  231. ```bash
  232. sqlite3 "$VDB_HOME/data.index.vdb7" ".schema cve_index"
  233. sqlite3 "$VDB_HOME/data.index.vdb7" "SELECT count(*) FROM cve_index;"
  234. sqlite3 "$VDB_HOME/data.index.vdb7" "SELECT count(*) FROM cve_metadata;"
  235. sqlite3 "$VDB_HOME/data.vdb7" "SELECT count(*) FROM cve_data;"
  236. sqlite3 "$VDB_HOME/data.vdb7" "SELECT count(*) FROM cve_source_data;"
  237. ```
  238. A healthy build has many more `cve_data` rows than `cve_source_data` rows,
  239. because a CVE affecting many packages shares one blob. In a default database
  240. `cve_metadata` and `cve_metadata_text` have zero rows; in an
  241. `--include-metadata` build they are populated.
  242. Open a database you did not build read-only:
  243. ```bash
  244. sqlite3 "file:$VDB_HOME/data.index.vdb7?mode=ro" "SELECT count(*) FROM cve_index;"
  245. ```
  246. Two traps when querying by hand. Locators are lowercased, so lowercase your
  247. query. And a bare `SELECT ... FROM cve_index` scans in name order via a
  248. covering index rather than by CVE id, so ordering by `cve_id` costs nothing but
  249. assuming an order costs correctness.
  250. Direct SQL sees none of the coverage, staleness or empty-database guards the
  251. library applies. On a shard, a purl of an uncovered type returns no rows, and
  252. that is indistinguishable in SQL from clean. Prefer the library APIs when the
  253. answer will be reported to someone.
  254. ## Skill: investigate a false positive or false negative
  255. Trace in order:
  256. 1. The source converter (`osv.py`, `aqua.py`, `gha.py`, `nvd.py`), including
  257. whether the record survived its feed's year floor at all.
  258. 2. `VulnerabilityDetail` fields: `mii`, `mie`, `mai`, `mae`, `fixed_location`,
  259. `package_type`.
  260. 3. `to_purl_vers()` in `utils.py`.
  261. 4. `CVESource.store5()`, for the purl prefix and index row it writes.
  262. 5. The search path and `vdb.lib.vers.vers_compare()`.
  263. For a false negative, check step 4 against step 5 specifically. The most common
  264. cause is a stored locator the lookup never proposes: both sides build a purl
  265. prefix independently, and when they disagree the rows exist and are
  266. unreachable, which looks exactly like "not vulnerable".
  267. Add the regression test at the lowest layer that demonstrates the bug.
README.md · README · 1,121 lines
  1. # appthreat-vulnerability-db
  2. An offline vulnerability database and package search library. `vdb` downloads
  3. data from AppThreat vuln-list, OSV, NVD and GitHub, converts it into the CVE 5.2
  4. schema, and stores it in SQLite with indexes built for fast offline lookup by
  5. purl, CPE or git URL.
  6. Every row uses an open specification, CVE 5.2 for the records and Package URL
  7. (purl and vers) for the locators, so the data stays readable without this
  8. library. The `.vdb7` files are ordinary SQLite databases.
  9. Upgrading from 6.x? Read [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) first. The
  10. database files, the published image URLs, three environment variables and the
  11. version comparison API all changed, and v7 does not read v6 databases.
  12. Measured v6-versus-v7 figures for build time, database size, query latency and
  13. shard footprint are in [docs/BENCHMARKS.md](./docs/BENCHMARKS.md).
  14. The full documentation, including a CLI reference, an environment variable
  15. index and hands-on tutorials, is published at
  16. [appthreat.github.io/vulnerability-db](https://appthreat.github.io/vulnerability-db/)
  17. from the [docs](./docs) directory.
  18. ## What it is for
  19. Advisories are merged from several upstream feeds because the same CVE is
  20. often published independently by NVD, an OSV feed and a GitHub advisory, with
  21. different affected ranges; querying the union of them reduces false negatives.
  22. Purl prefixes are generated even for git repositories, so a search can start
  23. from a purl, a CPE, or an HTTP git URL.
  24. Lookups run entirely against local SQLite files; there is no network access
  25. during a scan. Missing data is reported explicitly instead of silently: an
  26. absent or emptied database exits 1 with an error message, a purl type that no
  27. local shard serves raises `ShardCoverageError`, and batch searches mark
  28. uncovered components `coverage_gap` rather than returning an empty list.
  29. ```
  30. vdb --cache / --cache-os
  31. Upstream feeds -------------------------> data.vdb7 (CVE 5.2 records)
  32. data.index.vdb7 (purl/vers index)
  33. |
  34. optional split
  35. v
  36. shards/<type>/ (per-type databases)
  37. ```
  38. ## Data sources
  39. Linux [vuln-list](https://github.com/appthreat/vuln-list) (forked from Aqua
  40. Security), OSV, NVD and GitHub.
  41. Linux and OSS-Fuzz OSV feeds are excluded by default; set `OSV_INCLUDE_FUZZ=true`
  42. to include them. Malware feeds are included by default and increase the database
  43. size slightly; `OSV_EXCLUDE_MALWARE=true` excludes them, though a missed
  44. malicious package is a worse outcome than a larger artifact.
  45. Application ecosystem advisories are not filtered by year. `NVD_START_YEAR`
  46. bounds NVD-style CVE data and the Linux distro feeds only. The reason is that a
  47. distro fix moves the whole release forward, so an old distro advisory describes
  48. versions nobody runs, whereas a lockfile does not move, and `lodash@4.17.15` or
  49. `log4j-core@2.14.1` are in projects today.
  50. ### Linux distros
  51. AlmaLinux, Alpine, Amazon Linux, Arch Linux, Azure Linux (CBL-Mariner),
  52. Bottlerocket, Debian, Oracle Linux, RHEL/CentOS, Rocky Linux, Ubuntu, Photon,
  53. Chainguard, Wolfi OS, Mageia, Alpaquita, MinimOS, and SLES plus openSUSE Leap
  54. and Tumbleweed as opt-in feeds.
  55. A few details that affect how rows are stored and matched:
  56. Debian, Ubuntu, Red Hat, SUSE and openSUSE come from the corresponding OSV
  57. feeds, which carry per-release fixed versions and CVE ids. Red Hat and SUSE
  58. errata fan out to every CVE they name. SUSE family rows are stored under
  59. channel-scoped `pkg:rpm/suse|opensuse/<release>/<name>` locators that match the
  60. `distro` qualifier cdxgen emits. The equivalent vuln-list subtrees duplicated
  61. this data less precisely, or under locators no scanner emits, and are ignored at
  62. ingest.
  63. Azure Linux comes from the OSV `Azure Linux` feed, which publishes `AZL-*`
  64. advisories carrying one CVE each with per-release fixed versions. Rows use
  65. channel-scoped `pkg:rpm/azure-linux/<release>/<name>` locators, and the
  66. `azurelinux`, `cbl-mariner` and `mariner` purl namespaces cdxgen emits all alias
  67. onto `azure-linux`.
  68. Photon rows are release-scoped as `pkg:rpm/photon/photon-<release>/<name>`.
  69. Bottlerocket comes from the vuln-list `bottlerocket` tree (BRSA and
  70. GHSA-id'd updateinfo advisories). Rows are flat `pkg:rpm/bottlerocket/<name>`
  71. locators — there is no bottlerocket purl type in the spec, and cdxgen
  72. namespaces rpms by the os-release ID; a trivy-style `pkg:bottlerocket/<name>`
  73. scan purl is folded onto the same locator — with full version-release bounds
  74. and the CVE taken from the advisory references. Package entries whose release
  75. is a bare commit hash name the Bottlerocket OS release that shipped the fix,
  76. not a package version, and are skipped. `VDB_IGNORE_BOTTLEROCKET=true`
  77. excludes the tree.
  78. Oracle Linux comes from the vuln-list `oval/oracle` tree (ELSA errata, and
  79. ELBA errata that name CVEs). Rows are scoped by major release,
  80. `pkg:rpm/oracle/oracle-<N>/<name>` (module streams
  81. `oracle-<N>-stream<S>`): Oracle ships every fix in the latest repository, so
  82. a host on an older minor release still matches. The release comes from each
  83. bound's `el` tag (including `elNuek`) or from the `Oracle Linux N is
  84. installed` criterion of its own branch, Ksplice-only branches are skipped, and
  85. `distro=oracle-8.10`, `distro=ol-8.10` and the `pkg:rpm/ol/...` namespace all
  86. fold onto the same channel. Each row carries its own CVE's CVSS3 vector and
  87. metrics from the erratum. `VDB_IGNORE_ORACLE=true` excludes the tree.
  88. ### Seal Security
  89. The two `vulnfeed.sealsecurity.io` OSV feeds carry CVE advisories for the
  90. packages Seal rebuilt with backported fixes, under Seal's own names
  91. (`seal-requests`, `@seal-security/ajv`) and under the original names with
  92. Seal version markers (`requests@2.14.2+sp1`). Every stored range must sort
  93. above the upstream release it rebuilds under the ecosystem's own comparator;
  94. ranges that do not are dropped rather than widened, so upstream releases never
  95. match. Seal:PyPI ranges are kept, because the pypi comparator follows PEP 440 and
  96. sorts a local version such as `1.26.18+sp0` above the release `1.26.18`. A few
  97. Go `+incompatible` and Maven ranges are dropped. Seal's `+sp999` ceiling is reported as no
  98. fix. The feeds are registered in app-only builds too; those keep the
  99. application ecosystems and skip Seal's Debian, Ubuntu, Alpine and Red Hat
  100. rows. `VDB_IGNORE_SEAL=true` excludes both feeds.
  101. Chainguard, Wolfi and MinimOS advisories filed under a fix version of `0`
  102. are not stored. That is the Alpine secdb and OSV spelling of "not affected"
  103. (a false-positive determination), not "no fix yet".
  104. ## Installation
  105. ```shell
  106. pip install appthreat-vulnerability-db>=7.0.0
  107. ```
  108. For optional dependencies such as `oras`, install the `[oras]` or `[all]` group:
  109. ```shell
  110. pip install appthreat-vulnerability-db[all]
  111. ```
  112. Users of dep-scan v5 need the 5.8.x line instead:
  113. ```shell
  114. pip install appthreat-vulnerability-db==5.8.0
  115. ```
  116. ## Getting a database
  117. This package is primarily a library, used by
  118. [owasp-dep-scan](http://github.com/owasp-dep-scan/dep-scan). The CLI exists to
  119. test and explore it directly.
  120. > [!IMPORTANT]
  121. > The AppThreat-hosted database images and workflows are best treated as
  122. > bootstrap or evaluation defaults. For production, especially for the larger
  123. > app+OS variants, build and publish your own databases from your own CI.
  124. >
  125. > You control when data is built, where it is published, and which upstream
  126. > sources and retention windows are allowed. You can publish smaller artifacts
  127. > that match your environment instead of pulling a one-size-fits-all image. And
  128. > app+OS builds need significant compute, disk and network, which is easier to
  129. > budget for on infrastructure you own.
  130. ### Option 1: `vdb db` for staged, validated refreshes (preferred)
  131. `vdb db` is the front end for getting and refreshing data: the full image, a
  132. named set of type shards, or everything already present locally. It is the
  133. preferred way to obtain a database, because every refresh is staged, validated
  134. and swapped in atomically.
  135. ```shell
  136. vdb db status # what is local: main DB and per-shard build id, age, size, coverage
  137. vdb db refresh full # the full image, staged, validated and swapped
  138. vdb db refresh full --flavor app-extended # app-scope + metadata tables
  139. vdb db refresh npm pypi # named shards (shard names or purl types; nuget resolves to app)
  140. vdb db refresh --all # every shard the local build published (siblings.available)
  141. vdb db refresh # sync what I have: everything already in the local store
  142. vdb db refresh npm --tag v7-xz # a different release line (see below)
  143. ```
  144. Shard tags always carry an encoding suffix, because that is all the build
  145. publishes: `vdb7-npm:v7.0.x-xz` and `vdb7-npm:v7.0.x-zst` exist, a bare
  146. `vdb7-npm:v7.0.x` does not. `vdb db refresh` appends it for you, inheriting
  147. whichever encoding the local store was fetched with and defaulting to `xz`.
  148. `zst` works when a zstd decompressor is available — Python 3.14+ (which ships
  149. `compression.zstd`) or a `zstd` binary on PATH (`VDB_ZSTD_BIN` points at one
  150. elsewhere); without one, a zst refresh
  151. fails before downloading anything and says so. `xz` needs no decompressor,
  152. which is why it stays the default. Use `--tag` to pick a release line rather
  153. than to restore a suffix.
  154. The remaining refresh flags: `--registry` points shards at your own registry
  155. (default `ghcr.io/appthreat`, or the siblings record of a local shard),
  156. `--store-dir` overrides the shard store location (default `$VDB_SHARDS_DIR`),
  157. and `--compression xz` makes the encoding choice explicit. `refresh full`
  158. also accepts `--flavor <name>` to pick one of the published full-database
  159. variants (`full`, `app-only`, `app-extended`, `app-2y`, `app-2y-extended`,
  160. `app-10y`, `app-10y-extended`;
  161. see [Scope variants](#scope-variants)), `--app-only` as the shorthand for
  162. `--flavor app-only`, and `--image <oci-ref>` for a single-run URL override
  163. that wins over both.
  164. `vdb db status` reports, per shard, the build id, build time and age, sizes,
  165. type coverage, whether the store is full or partial (naming any missing group
  166. shards), whether it mixes builds, and which shards are much older than their
  167. siblings. Mixing builds is a normal state, because shards are independent
  168. artifacts. `--json` emits the same report machine-readably.
  169. Refreshes are safe to interrupt. Shard refreshes place each shard with an atomic
  170. directory rename. The full-image refresh downloads into a staging directory,
  171. validates the artifact (both `.vdb7` files present, index non-empty, not a
  172. single-shard artifact), and only then swaps it in, stashing the current files
  173. away before the new ones land. An interrupted refresh therefore leaves the main
  174. directory with files _missing_, which is loudly reported as an empty database
  175. with exit 1, rather than a mixed old and new pair that could answer subtly
  176. wrong. A failed or empty fetch is an error and never replaces a working
  177. database.
  178. If you use dep-scan: `vdb db` subsumes the mechanism `depscan-vdb` provides,
  179. which is downloading an image into `VDB_HOME`, fetching and refreshing shards,
  180. and reporting local state. Variant selection, retry policy and the freshness
  181. marker remain dep-scan's concerns. See
  182. [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) section 3.
  183. ### Option 2: download a pre-built database
  184. The app-only database (about 160 MB to download, around 1 GB on disk) is
  185. [refreshed](https://github.com/AppThreat/vdb/actions) every 12 hours and is the
  186. fastest way to evaluate `vdb`, bootstrap a workstation, or validate an
  187. integration.
  188. ```shell
  189. vdb db refresh full --app-only
  190. ```
  191. For container and OS scans, download the full image (about 730 MB to download,
  192. 15.6 GB on disk):
  193. ```shell
  194. vdb db refresh full
  195. ```
  196. To consume an artifact your own workflow published, override the URL:
  197. ```shell
  198. export VDB_DATABASE_URL=your.registry/vdb7-full:your-tag-xz
  199. vdb db refresh full
  200. ```
  201. Keep the `-xz` suffix on any URL you set: it unpacks with no further
  202. requirements. A `-zst` URL also works, but needs a zstd decompressor
  203. (Python 3.14+, or a `zstd` binary on PATH); without one the download is
  204. refused up front with both remedies named.
  205. Metadata searches (full text, alias, reference, package name, symbol, source,
  206. severity, date and malware-aware filters) need a database whose metadata tables
  207. are populated. The v7 build publishes those as the extended flavors —
  208. app-scope, NVD from 2020 or 2016:
  209. ```shell
  210. vdb db refresh full --flavor app-extended # NVD 2020+ + metadata tables
  211. ```
  212. Use `--flavor app-10y-extended` for metadata plus NVD back to 2016. For any
  213. other scope, build one yourself:
  214. ```shell
  215. vdb --cache --include-metadata
  216. ```
  217. ### Option 3: ORAS
  218. ```shell
  219. export VDB_HOME=$HOME/vdb
  220. oras pull ghcr.io/appthreat/vdb7-full:v7.0.x-xz -o $VDB_HOME
  221. tar -xvf *.tar.xz
  222. rm *.tar.xz
  223. ```
  224. Use `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` for the app-only database, or
  225. `ghcr.io/appthreat/vdb7-<shard>:v7.0.x-xz` for a single type shard. Note the
  226. `-only`: `vdb7-app` is the shard store's app group shard, a partial slice of
  227. the full database, not a database you install on its own.
  228. `vdb db refresh` does this for you and validates what it downloads. The `-zst`
  229. artifacts published alongside each `-xz` one are smaller to download; vdb can
  230. fetch them too when a zstd decompressor is present (Python 3.14+ or a `zstd`
  231. binary), and manual pipelines can always unpack zstd themselves.
  232. ### Option 4: Hugging Face
  233. The v7 artifacts live under `v7-<artifact>/` prefixes of the
  234. [AppThreat/vdb](https://huggingface.co/datasets/AppThreat/vdb) dataset:
  235. `v7-full/`, `v7-app/`, `v7-npm/` and so on.
  236. ```shell
  237. pip install -U "huggingface_hub[cli]"
  238. export VDB_HOME=$(pwd)/app
  239. hf download --repo-type dataset AppThreat/vdb v7-app/data.vdb7.zst --local-dir .
  240. ```
  241. Uncompressed `.vdb7` files are on Hugging Face only, and only for the artifacts
  242. small enough to be worth it. `full`, `deb`, `rpm` and `apk` are published
  243. compressed and are best fetched over ORAS with `vdb db refresh`.
  244. ### Option 5: build your own (recommended for production)
  245. If you depend on `vdb` regularly, build and publish your own databases
  246. internally. This is the recommended approach for enterprises, security teams
  247. and integrators: you publish from infrastructure you trust, you reduce
  248. supply-chain and availability dependencies on third-party refresh jobs, you can
  249. tune the scope to your environment, and you can put larger app+OS builds on
  250. runners sized for them.
  251. The workflow is to set your retention and distro selection variables, run
  252. `vdb --cache` or `vdb --cache-os` on a schedule, package the resulting `.vdb7`
  253. files, publish them to your own registry or object store, and point clients at
  254. your URL.
  255. ```shell
  256. vdb --cache # application vulnerabilities, around 1 GB
  257. vdb --cache --include-metadata # the same, with metadata search tables
  258. vdb --cache-os # application and OS vulnerabilities, around 15.6 GB
  259. vdb --cache-os --include-metadata
  260. vdb --cache --only-osv # OSV as the only source
  261. vdb --clean # remove existing databases
  262. ```
  263. `--include-metadata` is equivalent to `VDB_INCLUDE_METADATA=true`. Builds print
  264. minimal progress, such as the source year being fetched or the latest CVE
  265. stored; `--quiet` suppresses the logo, logs and progress.
  266. Two knobs change coverage rather than just size:
  267. ```shell
  268. NVD_START_YEAR # default 2022, supports back to 2002. NVD-style data and distro feeds only.
  269. GITHUB_PAGE_COUNT # default 2, supports up to 20
  270. ```
  271. A team that only scans modern application dependencies can build a much smaller
  272. artifact with a recent `NVD_START_YEAR`, noting that
  273. `VDB_APP_ECOSYSTEM_START_YEAR` is unset by default so library advisories are
  274. kept regardless of age. A platform team supporting a subset of distros can use
  275. the `VDB_IGNORE_*` or `VDB_INCLUDE_*` variables before `vdb --cache-os`:
  276. ```shell
  277. export VDB_IGNORE_ALMALINUX=true
  278. export VDB_IGNORE_UBUNTU=true
  279. vdb --cache-os
  280. ```
  281. See `LINUX_DISTRO_VULN_LIST_PATHS` in [config.py](./vdb/lib/config.py) for the
  282. full list of distro strings.
  283. Set `VDB_TEMP_DIR` to a partition with room. An app+OS build needs significant
  284. temporary space for index creation and `VACUUM`, and a small `/tmp` is the most
  285. common build failure.
  286. ## The two database files
  287. **`data.index.vdb7`** is the search side: small, random-access, hot. It holds
  288. `cve_index`, one row per `(cve_id, vers, purl_prefix)`, plus the optional
  289. `cve_metadata` and `cve_metadata_text` tables.
  290. **`data.vdb7`** is the payload side: large, sequential, cold. It holds
  291. `cve_source_data`, where each CVE 5.2 source blob is stored once and keyed by
  292. hash, and `cve_data`, the package locator rows that reference those blobs.
  293. The split follows the access patterns. You resolve a package against the index
  294. first, which is cheap, and hydrate only what matched. A CVE affecting 300
  295. packages costs one blob rather than 300 copies.
  296. Default public databases leave `cve_metadata` and `cve_metadata_text` empty to
  297. minimise index size, and v7 publishes no artifact with them populated. Build
  298. with `--include-metadata` for text, alias, reference, symbol, severity, source
  299. and date-aware searches, or publish your own metadata artifact and point
  300. `VDB_DATABASE_URL` at it. Public artifacts are rebuilt from scratch by the
  301. release workflows, so migrations for older `.vdb6` files are not maintained.
  302. Any SQLite browser or CLI can read both files. See
  303. [INTEGRATION.md](./INTEGRATION.md) for the schemas and query patterns.
  304. ## Type-shard databases
  305. The full, app-only and per-type shard images are all published and usable
  306. today: `ghcr.io/appthreat/vdb7-full:v7.0.x-xz`,
  307. `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` and
  308. `ghcr.io/appthreat/vdb7-<type>:v7.0.x-xz` for every shard the build produced,
  309. plus `-zst` variants and the floating `v7-*` and `latest-*` tags. A shard's own
  310. `vdb.meta` (`artifact.types` and `siblings.available`) is the authority on what
  311. exists, because the shard set grows when a new ecosystem first gets an
  312. advisory; check a specific artifact with
  313. `oras manifest fetch <ref> --descriptor`.
  314. A shard is a standalone database holding only the purl types it declares, so a
  315. scanner that only sees npm packages can download a fraction of the full
  316. artifact and get byte-identical answers for those types. On disk, a `pypi`-only
  317. workload needs about 90 MB of shard (82.7 MB data plus 7.4 MB index) instead
  318. of the 15.6 GB full production pair, a roughly 170-fold reduction in
  319. uncompressed bytes. Both figures are uncompressed on-disk sizes; the published
  320. images are xz- or zst-compressed and the compressed ratio differs, so compare
  321. download sizes against download sizes rather than mixing the two.
  322. Five shards are structural: `deb`, `rpm`, `apk`, `app` (the application
  323. ecosystem union) and `cpe`, which is the complement holding every purl-prefix
  324. type no other shard claims. The rest are per-type, one for each
  325. [purl-spec type](https://github.com/package-url/purl-spec/tree/main/types)
  326. except `deb`, `rpm`, `apk` and `generic`. That is 43 possible shard names, and
  327. the splitter emits only the ones the build actually has rows for, which was 27
  328. on the 2026-08-23 app+OS build. A shard's own `vdb.meta` (`artifact.types` and
  329. `siblings.available`) is the authority on what exists and what it serves, so do
  330. not hard-code a shard list. Purl types the spec does not define stay in `cpe`.
  331. Per-type shards deliberately overlap the group shards rather than partitioning
  332. them: the `npm` shard and the `app` shard both serve npm. They are separate
  333. views, which is why removing a type from `cpe` would strand it for a store that
  334. only carries the group shards.
  335. To use shards, point your download flow at a shard artifact, which becomes the
  336. connected main database under `VDB_HOME` with `completeness: "partial"` in its
  337. `vdb.meta`, or fetch shards into the store with `vdb db refresh`. Additional
  338. shards go under `$VDB_SHARDS_DIR` (default `$VDB_HOME/shards`).
  339. Purl searches (`search_by_purl_like`, `search_packages_batched`,
  340. `search_bom_*`) fan out across the local shards automatically. CVE, alias, CPE
  341. and text searches require the full database and raise on a shard rather than
  342. under-report.
  343. Shard-mode results carry a coverage companion (`.coverage` on the returned
  344. list, `vdb.lib.search.last_shard_coverage`, and a `coverage` key on BOM
  345. results) reporting which purl types were checked. Uncovered types are marked
  346. `coverage_gap` and warn on stderr, so a partial scan cannot look clean. A shard
  347. more than 7 days older than its siblings triggers a staleness warning at search
  348. time, and a _missing_ shard stays a distinct condition (a coverage gap, or
  349. `ShardCoverageError`) rather than being collapsed into staleness. Set
  350. `VDB_AUTO_FETCH=true` to let searches fetch missing shards on demand, or call
  351. `vdb.lib.shard_store.fetch_shards()` explicitly.
  352. See `contrib/shard_store.md` for the design, measurements and safety argument, and
  353. `contrib/shard_split.md` for the splitter.
  354. ### Automatic shard splitting
  355. After a successful `vdb --cache` or `vdb --cache-os` build, the database is
  356. split by purl type into the shards directory. Splitting runs only when the build
  357. completed and the result is complete (`completeness: full` in `vdb.meta`), so a
  358. partial or failed build never produces a shard set. Shards that would be empty
  359. for your configuration, such as the OS distro shards of an application-only
  360. build, are skipped. If the split fails, the failure is reported and the
  361. partially written shards are discarded, while the full database that was just
  362. built stays intact and usable.
  363. ```shell
  364. vdb --cache --no-split
  365. ```
  366. `--no-split` is honoured by every entry point that triggers a build: the `vdb`
  367. console script, `python -m vdb.cli`, and library calls to `vdb.cli.main()`.
  368. The splitter is also available for re-splitting an existing database or
  369. verifying a shard set. The source database is opened read-only.
  370. ```shell
  371. vdb split --db-file ~/.vdb/data.vdb7 --index-file ~/.vdb/data.index.vdb7 -o /path/to/shards -v
  372. vdb verify --db-file ~/.vdb/data.vdb7 --index-file ~/.vdb/data.index.vdb7 --shards-root /path/to/shards
  373. ```
  374. `vdb split` accepts `--shards`, `--build-id`, `--version-tag`, `--jobs`
  375. (`--jobs 0` uses one worker per CPU) and `--report` to write a split report
  376. JSON. `vdb verify` checks that the group shards exactly partition the full
  377. database and exits non-zero when they do not.
  378. ## Published artifacts and their sizes
  379. The shard set is not fixed: the splitter emits one `vdb7-<type>` artifact for
  380. every purl type the build actually has rows for, so it grows when a new
  381. ecosystem first gets an advisory (this build produced 27). The full artifact's
  382. `vdb.meta` (`siblings.available`) is the authority on what exists.
  383. If you are choosing automatically, pick the smallest row whose types cover the
  384. SBOM you are scanning. `full` is only needed for CPE lookups or when you cannot
  385. predict the types in advance.
  386. | Artifact | Purl types served | Rows | Uncompressed data | Uncompressed index | `.tar.xz` | `.zst` |
  387. | :----------------- | :------------------------------------------------------------------------------ | --------: | ----------------: | -----------------: | --------: | ------: |
  388. | `vdb7-full` | everything, including CPE | 6,894,338 | 12.2 GiB | 2.3 GiB | 733 MB | 806 MB |
  389. | `vdb7-deb` | `deb` | 2,995,928 | 4,139 MB | 928 MB | 153 MB | 166 MB |
  390. | `vdb7-rpm` | `rpm` | 2,084,872 | 6,171 MB | 977 MB | 235 MB | 271 MB |
  391. | `vdb7-apk` | `apk` | 1,137,398 | 1,294 MB | 359 MB | 104 MB | 111 MB |
  392. | `vdb7-app` | `npm`, `pypi`, `maven`, `golang`, `nuget`, `gem`, `composer`, `cargo`, `github` | 396,377 | 923 MB | 109 MB | 161 MB | 170 MB |
  393. | `vdb7-cpe` | the complement: 35 types no other shard claims, incl. `generic` | 279,763 | 552 MB | 89 MB | 48.8 MB | 51.7 MB |
  394. | `vdb7-npm` | `npm` | 237,664 | 509 MB | 56 MB | 50.0 MB | 52.7 MB |
  395. | `vdb7-github` | `github` | 63,122 | 161 MB | 23 MB | 20.3 MB | 21.6 MB |
  396. | `vdb7-pypi` | `pypi` | 31,823 | 82.7 MB | 7.4 MB | 8.2 MB | 8.8 MB |
  397. | `vdb7-maven` | `maven` | 20,225 | 43.0 MB | 7.7 MB | 4.1 MB | 4.5 MB |
  398. | `vdb7-composer` | `composer` | 17,178 | 44.1 MB | 5.7 MB | 4.3 MB | 4.7 MB |
  399. | `vdb7-golang` | `golang` | 8,458 | 45.4 MB | 3.3 MB | 4.8 MB | 5.3 MB |
  400. | `vdb7-nuget` | `nuget` | 8,040 | 20.5 MB | 3.1 MB | 1.1 MB | 1.2 MB |
  401. | `vdb7-gem` | `gem` | 6,138 | 12.4 MB | 1.4 MB | 1.2 MB | 1.3 MB |
  402. | `vdb7-cargo` | `cargo` | 3,729 | 12.4 MB | 1.1 MB | 1.4 MB | 1.5 MB |
  403. | `vdb7-julia` | `julia` | 2,040 | 5.7 MB | 0.6 MB | 610 KB | 635 KB |
  404. | `vdb7-alpm` | `alpm` | 1,500 | 2.0 MB | 0.4 MB | 175 KB | 185 KB |
  405. | `vdb7-hex` | `hex` | 293 | 1.3 MB | 0.1 MB | 162 KB | 170 KB |
  406. | `vdb7-swift` | `swift` | 75 | 0.7 MB | 0.1 MB | 67 KB | 71 KB |
  407. | `vdb7-docker` | `docker` | 74 | 0.3 MB | 0.1 MB | 23 KB | 24 KB |
  408. | `vdb7-opam` | `opam` | 61 | 0.3 MB | 0.1 MB | 31 KB | 33 KB |
  409. | `vdb7-bitnami` | `bitnami` | 41 | 0.2 MB | 0.1 MB | 5.1 KB | 4.9 KB |
  410. | `vdb7-huggingface` | `huggingface` | 40 | 0.2 MB | – | 16 KB | 17 KB |
  411. | `vdb7-pub` | `pub` | 28 | 0.2 MB | – | 15 KB | 16 KB |
  412. | `vdb7-bitbucket` | `bitbucket` | 26 | 0.2 MB | – | 12 KB | 13 KB |
  413. | `vdb7-cran` | `cran` | 13 | 0.1 MB | – | 7.6 KB | 7.6 KB |
  414. | `vdb7-cocoapods` | `cocoapods` | 6 | 0.1 MB | – | 4.4 KB | 4.2 KB |
  415. | `vdb7-conda` | `conda` | 2 | 0.1 MB | – | 3.0 KB | 2.7 KB |
  416. Use `vdb7-full` for container and VM scanning and for anything needing CPE,
  417. CVE-ID, alias, reference or full-text search. Use `vdb7-app-only` when you
  418. never scan OS packages. Use a type shard when your workload is scoped to one
  419. ecosystem.
  420. `vdb7-app` and `vdb7-app-only` are not the same artifact and are not
  421. interchangeable. `vdb7-app` is the group shard above: `completeness=partial`,
  422. for the shard store, placed with `vdb db refresh app`. `vdb7-app-only` is a
  423. complete database built from an app-only ingest: `completeness=full`,
  424. installed as the main database with `vdb db refresh full --app-only`. Pointing
  425. `refresh full` at `vdb7-app` is rejected by design.
  426. Every shard in this build passed the match-set gate, meaning `CLEAN` on the
  427. corpus, the prefix probes and the absent-type checks: a shard returns exactly
  428. what the full database returns for the types it serves.
  429. ### Artifact URLs
  430. Every artifact is published to GHCR as an OCI image carrying the two database
  431. layers plus a `vdb.meta` config, and mirrored to the `AppThreat/vdb` Hugging
  432. Face dataset under a `v7-<artifact>/` directory.
  433. ```text
  434. ghcr.io/appthreat/vdb7-full:v7.0.x-xz app+OS, everything incl. CPE
  435. ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz application ecosystems only
  436. ghcr.io/appthreat/vdb7-app-extended:v7.0.x-xz + metadata tables
  437. ghcr.io/appthreat/vdb7-app-2y:v7.0.x-xz NVD from 2024
  438. ghcr.io/appthreat/vdb7-app-2y-extended:v7.0.x-xz NVD from 2024 + metadata
  439. ghcr.io/appthreat/vdb7-app-10y:v7.0.x-xz NVD from 2016
  440. ghcr.io/appthreat/vdb7-app-10y-extended:v7.0.x-xz NVD from 2016 + metadata
  441. ghcr.io/appthreat/vdb7-app:v7.0.x-xz app group SHARD (partial)
  442. ghcr.io/appthreat/vdb7-npm:v7.0.x-xz one purl type
  443. ghcr.io/appthreat/vdb7-pypi:v7.0.x-xz
  444. hf download --repo-type dataset AppThreat/vdb v7-app-only/data.vdb7.zst
  445. hf download --repo-type dataset AppThreat/vdb v7-npm/data.vdb7 # raw .vdb7
  446. ```
  447. `v7-xz` and `v7-zst` track the newest build, `latest-xz` and `latest-zst` are
  448. aliases for them, and `v7.0.x-xz` and `v7.0.x-zst` pin the release line. Swap
  449. the suffix for `-zst` to fetch the zstd variant. On Hugging Face, `full`,
  450. `deb`, `rpm`, `apk`, `app-10y` and `app-10y-extended` are published compressed
  451. only; smaller artifacts are also mirrored as raw `.vdb7` files.
  452. ```shell
  453. oras pull ghcr.io/appthreat/vdb7-app-only:v7-xz # newest build
  454. oras pull ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz # pinned release line
  455. ```
  456. ### Scope variants
  457. Beyond `vdb7-full` and `vdb7-app-only`, the build publishes five app-scope
  458. variants that differ only in how much they ingest. All are complete databases
  459. (`completeness=full`), installable as the main database. `--flavor` names are
  460. the published repo suffixes, so `--flavor app-extended` installs
  461. `ghcr.io/appthreat/vdb7-app-extended:v7.0.x-xz` (or your mirror's equivalent:
  462. the non-env flavors inherit their registry and tag from `VDB_DATABASE_URL`).
  463. | Artifact | NVD from | Metadata tables | Install with |
  464. | :---------------------- | :------- | :-------------- | :---------------------------------------------- |
  465. | `vdb7-app-only` | 2020 | no | `vdb db refresh full --app-only` |
  466. | `vdb7-app-extended` | 2020 | yes | `vdb db refresh full --flavor app-extended` |
  467. | `vdb7-app-2y` | 2024 | no | `vdb db refresh full --flavor app-2y` |
  468. | `vdb7-app-2y-extended` | 2024 | yes | `vdb db refresh full --flavor app-2y-extended` |
  469. | `vdb7-app-10y` | 2016 | no | `vdb db refresh full --flavor app-10y` |
  470. | `vdb7-app-10y-extended` | 2016 | yes | `vdb db refresh full --flavor app-10y-extended` |
  471. `--image <oci-ref>` still overrides the URL for any single refresh, e.g. to
  472. pin a different release line: `vdb db refresh full --flavor app-10y --image
  473. ghcr.io/appthreat/vdb7-app-10y:v7-xz`.
  474. "Metadata tables" means the build ran with `--include-metadata`, so the
  475. metadata search APIs have data to answer from. Without them those APIs return
  476. nothing rather than failing, which is why the `-extended` variants exist: they
  477. are the pre-built answer to "build locally with `--include-metadata`".
  478. The 10-year variants cover NVD back to 2016 instead of 2020. That widens
  479. CVE-ID and CPE coverage for older advisories; it does not change how OSV
  480. ecosystem data is ingested.
  481. There is no `vdb7-app-2y` on the v7 line yet, and no OS-inclusive `-extended`
  482. or `-10y` variant — the 6.7.x line published both. Only the app-scope variants
  483. above are built for v7.
  484. If you run your own workflow, keep the same naming pattern internally if it
  485. helps downstream tooling, but publish from infrastructure you control.
  486. ## CLI usage
  487. The CLI is flag-based, with three leading-token subcommands: `vdb split` and
  488. `vdb verify` for the splitter, and `vdb db` for getting and refreshing data.
  489. ```text
  490. usage: vdb [-h] [--clean] [--cache] [--cache-os] [--only-osv] [--only-aqua]
  491. [--only-ghsa] [--include-metadata] [--no-split] [--quiet]
  492. [--search SEARCH] [--search-text SEARCH_TEXT]
  493. [--search-alias SEARCH_ALIAS] [--search-reference SEARCH_REFERENCE]
  494. [--search-package-name SEARCH_PACKAGE_NAME]
  495. [--search-symbol SEARCH_SYMBOL] [--search-packages SEARCH_PACKAGES]
  496. [--batch-size BATCH_SIZE] [--list-malware] [--bom BOM_FILE]
  497. [--print-vdb-metadata] [--custom-data CUSTOM_DATA]
  498. options:
  499. -h, --help show this help message and exit
  500. --clean Clear the vulnerability database cache from platform
  501. specific user_data_dir.
  502. --cache Cache vulnerability information in platform specific
  503. user_data_dir.
  504. --cache-os Cache OS vulnerability information in platform specific
  505. user_data_dir.
  506. --only-osv Use only OSV as the source. Use with --cache.
  507. --only-aqua Use only Aqua vuln-list as the source. Use with --cache.
  508. --only-ghsa Use only recent ghsa as the source. Use with --cache.
  509. --include-metadata Populate extended metadata tables for text, alias,
  510. reference, symbol, severity, and source searches.
  511. Increases index database size.
  512. --no-split Skip splitting the built database into per-type shards
  513. under the shards directory (VDB_SHARDS_DIR). Splitting
  514. runs by default after a successful --cache/--cache-os
  515. build.
  516. --quiet Suppress logo, logs, and cache progress output.
  517. --search SEARCH Search for the package or vulnerability ID (CVE, GHSA,
  518. ALSA, DSA, etc.) in the database. Use purl, cpe, or git
  519. http url.
  520. --search-text SEARCH_TEXT
  521. Perform metadata/full-text search across vulnerability
  522. descriptions, aliases, references, and affected symbols.
  523. --search-alias SEARCH_ALIAS
  524. Search vulnerability aliases such as GHSA, OSV, or
  525. vendor advisory identifiers.
  526. --search-reference SEARCH_REFERENCE
  527. Search reference URLs and reference text in
  528. vulnerability metadata.
  529. --search-package-name SEARCH_PACKAGE_NAME
  530. Search vulnerability metadata by package name or
  531. namespace/name.
  532. --search-symbol SEARCH_SYMBOL
  533. Search affected functions or modules captured in
  534. vulnerability metadata.
  535. --search-packages SEARCH_PACKAGES
  536. Path to a JSON file containing a list of package
  537. locators to search in bulk. Each item may include purl,
  538. cpe, url, alias, package_name, or search.
  539. --batch-size BATCH_SIZE
  540. Batch size to use with --search-packages.
  541. --list-malware List latest malwares with CVE ID beginning with MAL-.
  542. --bom BOM_FILE Search for packages in the CycloneDX BOM file.
  543. --print-vdb-metadata Display metadata about the current vdb in
  544. user_data_dir.
  545. --custom-data CUSTOM_DATA
  546. Path to directory containing custom vulnerability data
  547. (JSON/YAML/TOML) to override/augment results.
  548. ```
  549. ### Searching
  550. ```shell
  551. vdb --search pkg:pypi/xml2dict@0.2.2
  552. vdb --search pkg:pypi/xml2dict # a purl prefix
  553. vdb --search "pkg:swift/github.com/vapor/vapor@4.39.0"
  554. vdb --search "pkg:swift/vapor/vapor@4.89.0" # short form
  555. vdb --search "cpe:2.3:a:npm:gitblame:*:*:*:*:*:*:*:*"
  556. vdb --search "npm:gitblame:0.0.1" # colon-separated
  557. vdb --search CVE-2024-25169
  558. vdb --search CVE-2025-% # wildcard
  559. vdb --search "https://github.com/electron/electron" # git url
  560. vdb --bom bom.json # CycloneDX SBOM
  561. ```
  562. Metadata searches need a database built with `--include-metadata`:
  563. ```shell
  564. vdb --search-text "deserialization parser"
  565. vdb --search-alias GHSA-gh-9999
  566. vdb --search-reference github.com/example/widget-ui
  567. vdb --search-package-name widget-ui
  568. vdb --search-symbol parse_payload
  569. ```
  570. Bulk search from a JSON file, optionally streamed in batches:
  571. ```shell
  572. vdb --search-packages packages.json
  573. vdb --search-packages packages.json --batch-size 25
  574. ```
  575. ```json
  576. [
  577. { "purl": "pkg:pypi/requests@2.31.0" },
  578. { "url": "https://github.com/pallets/flask" },
  579. { "alias": "GHSA-xxxx-yyyy-zzzz" },
  580. { "package_name": "flask" }
  581. ]
  582. ```
  583. To list malware entries:
  584. ```shell
  585. vdb --list-malware
  586. ```
  587. A search against an empty database exits 1 with an explicit message rather than
  588. printing no results. Wire CI to treat that as scanner down, not as clean.
  589. ## Custom vulnerability data
  590. VDB can load custom data from a local directory at runtime, to add private
  591. vulnerabilities that are not public, or to override false positives by marking
  592. specific versions as unaffected. Data must follow the CVE 5.2 JSON schema, in
  593. `.json`, `.yaml`, `.yml` or `.toml`.
  594. ```shell
  595. vdb --search pkg:npm/my-lib@1.0.0 --custom-data /path/to/custom/vulns
  596. ```
  597. Custom data is operator-controlled policy input. It can override official
  598. results, so in a hosted or automated integration treat the directory as a
  599. sensitive input path.
  600. ### Adding a private vulnerability
  601. Use the `cna` container, since you are defining a new record.
  602. ```yaml
  603. dataType: CVE_RECORD
  604. dataVersion: "5.2"
  605. cveMetadata:
  606. cveId: PRIVATE-2025-001
  607. assignerOrgId: 00000000-0000-4000-8000-000000000000
  608. state: PUBLISHED
  609. datePublished: "2025-01-01T00:00:00Z"
  610. dateUpdated: "2025-01-01T00:00:00Z"
  611. containers:
  612. cna:
  613. providerMetadata:
  614. orgId: 00000000-0000-4000-8000-000000000000
  615. descriptions:
  616. - lang: en
  617. value: "Private vulnerability in internal library"
  618. affected:
  619. - vendor: internal
  620. product: my-lib
  621. packageName: my-lib
  622. packageURL: pkg:npm/my-lib
  623. versions:
  624. - version: "1.0.0"
  625. status: affected
  626. versionType: semver
  627. lessThan: "2.0.0"
  628. ```
  629. ### Overriding a false positive
  630. Use an ADP (Authorized Data Publisher) container to append to or dispute
  631. existing data. When a CVE ID and purl combination exists in your custom data,
  632. VDB ignores the official entry and uses yours.
  633. ```yaml
  634. dataType: CVE_RECORD
  635. dataVersion: "5.2"
  636. cveMetadata:
  637. cveId: CVE-2023-9999
  638. assignerOrgId: 00000000-0000-4000-8000-000000000000
  639. state: PUBLISHED
  640. containers:
  641. adp:
  642. - providerMetadata:
  643. orgId: 00000000-0000-4000-8000-000000000000
  644. shortName: "MySecTeam"
  645. descriptions:
  646. - lang: en
  647. value: "Override to mark specific version as unaffected"
  648. affected:
  649. - product: requests
  650. packageName: requests
  651. packageURL: pkg:pypi/requests
  652. versions:
  653. - version: "2.31.0"
  654. status: unaffected
  655. versionType: semver
  656. ```
  657. ## Using the library
  658. `vulnerability-db` is designed to be embedded in Python applications. See
  659. [INTEGRATION.md](./INTEGRATION.md) for schemas, direct-SQL patterns and the
  660. full result shape.
  661. ```python
  662. import json
  663. import os
  664. from vdb.lib import config, db as db_lib, search
  665. from vdb.lib.orasclient import download_image
  666. # Prefer an organization-controlled URL. The upstream default is for local
  667. # testing and bootstrap.
  668. DB_URL = os.getenv("VDB_APP_ONLY_DATABASE_URL", config.VDB_APP_ONLY_DATABASE_URL)
  669. if db_lib.needs_update(days=1):
  670. download_image(DB_URL, config.DATA_DIR)
  671. # Optional: override results with local YAML/JSON definitions
  672. # search.load_custom_data("/path/to/custom_vulns")
  673. results = search.search_by_any("pkg:pypi/xml2dict@0.2.2", with_data=True)
  674. output = []
  675. for res in results:
  676. vuln = {"cve_id": res["cve_id"], "fixed_in": res["fix_version"]}
  677. if res.get("source_data"):
  678. # source_data is a plain dict, ready for json.dumps
  679. vuln["cve_data"] = res["source_data"]
  680. output.append(vuln)
  681. print(json.dumps(output, indent=2))
  682. ```
  683. `source_data` is a dict, not a Pydantic model, and has no `.model_dump()`.
  684. Snippets from the 6.x era that call
  685. `res["source_data"].model_dump(mode="json")` raise `AttributeError` on v7.
  686. For production, point `VDB_APP_ONLY_DATABASE_URL` or `VDB_DATABASE_URL` at your
  687. own workflow's artifacts so application instances do not depend on a
  688. third-party refresh job for security-relevant data.
  689. ### Bulk and filtered searches
  690. ```python
  691. from vdb.lib import search
  692. filters = {
  693. "severity_threshold": "HIGH",
  694. "sources": ["osv", "github"],
  695. "exclude_malware": True,
  696. "package_ecosystem": "pypi",
  697. "page_size": 25,
  698. }
  699. package_results = search.search_packages(
  700. [
  701. {"purl": "pkg:pypi/requests@2.31.0"},
  702. {"url": "https://github.com/pallets/flask"},
  703. {"cpe": "cpe:2.3:a:npm:lodash:4.17.20:*:*:*:*:*:*:*"},
  704. ],
  705. with_data=False,
  706. filters=filters,
  707. )
  708. for pkg in package_results:
  709. print(pkg["locator"], pkg["result_count"], pkg["max_severity"])
  710. ```
  711. For very large package sets, stream them:
  712. ```python
  713. for batch in search.search_packages_batched(packages, batch_size=100, with_data=False):
  714. for pkg in batch:
  715. print(pkg["locator"], pkg["result_count"])
  716. ```
  717. CycloneDX BOMs can be summarised or expanded:
  718. ```python
  719. summary = search.search_bom_summary("bom.json", filters={"severity_threshold": "MEDIUM"})
  720. detailed = search.search_bom_detailed("bom.json", with_data=True)
  721. ```
  722. For large SBOMs, `search_by_cdx_bom` yields batches so peak memory stays
  723. bounded:
  724. ```python
  725. for batch in search.search_by_cdx_bom("bom.json", with_data=True):
  726. for res in batch:
  727. ...
  728. ```
  729. ### Text, alias, reference and symbol search
  730. ```python
  731. search.search_by_alias("GHSA-xxxx-yyyy-zzzz", with_data=False)
  732. search.search_by_reference("github.com/pallets/flask", with_data=False)
  733. search.search_by_package_name("flask", with_data=False)
  734. search.search_by_symbol("parse_payload", with_data=False)
  735. search.search_full_text("deserialization parser", with_data=False)
  736. ```
  737. These use the metadata tables in `data.index.vdb7`. Locator searches work
  738. without metadata, so choose a `--include-metadata` database when your
  739. integration depends on these APIs or on metadata filters. On a database without
  740. metadata they fail safe by returning no metadata matches, which means an empty
  741. result there tells you nothing about the package.
  742. ### Partial (shard) databases
  743. A database whose `vdb.meta` declares `"completeness": "partial"` is a per-type
  744. shard. Purl lookups work normally and return exactly the full database's
  745. matches for the purl types the shard serves. Entry points whose queries span
  746. purl types (`search_by_cpe_like`, `search_by_cve`, `search_by_alias`,
  747. `search_by_reference`, `search_full_text`, `search_by_symbol`,
  748. `latest_malware`) raise `vdb.lib.search.PartialDatabaseError` rather than
  749. silently under-report.
  750. For a purl type no local shard serves, `search_by_purl_like` raises
  751. `vdb.lib.search.ShardCoverageError` instead of returning an empty list that
  752. would read as "no vulnerabilities". Batch and BOM searches instead mark the
  753. component `coverage_gap: true` and report a `coverage` block, so one missing
  754. shard does not fail a whole scan.
  755. ### Deployment notes
  756. Set `VDB_HOME` before importing `vdb.lib.config`. Path and build-scope
  757. variables are read at import time, so setting them later in a long-running
  758. process has no effect.
  759. For read-only deployments where the `.vdb7` files are not modified while the
  760. process runs, set `VDB_SQLITE_IMMUTABLE=true`.
  761. ## Environment variables
  762. Boolean toggles treat `true` or `1` as enabled. Set path and build-scope
  763. variables before importing `vdb.lib.config`.
  764. ### Paths and local storage
  765. | Variable | Default | Used by | Description |
  766. | :----------------- | :------------------------------- | :----------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  767. | `VDB_HOME` | Platform user data directory | CLI and library | Directory containing `data.vdb7`, `data.index.vdb7` and `vdb.meta`. |
  768. | `VDB_CACHE` | Platform user cache directory | `vdb --cache-os`, `AquaSource` | Cache directory. If `$VDB_CACHE/vuln-list.zip` exists it is used instead of downloading. |
  769. | `VDB_TEMP_DIR` | System temp directory | SQLite setup | Directory for SQLite temporary files during large builds and `VACUUM`. Use a partition with room for app+OS builds. |
  770. | `VDB_SHARDS_DIR` | `$VDB_HOME/shards` | Shard-mode search | Local type-shard store: one `<shard>/` directory per shard, each with `data.vdb7`, `data.index.vdb7` and a v2 `vdb.meta`. The connected main database participates as a shard when its own meta declares `completeness: "partial"`. |
  771. | `VDB_AUTO_FETCH` | unset | Shard-mode search | Opt-in on-demand shard fetch. When enabled, a shard-mode purl search needing a missing shard fetches it from the registry recorded in a local shard's `vdb.meta` siblings. Off by default so scans never reach for the network implicitly. |
  772. | `VDB_SHARD_FANOUT` | `sequential` | Shard-mode search | Multi-shard fan-out strategy. `sequential` runs the canonical per-shard search path; `attach` UNION-ALLs attached shard indexes for the index sweep. Both are gated by the multi-shard match-set gate. |
  773. | `VDB_ZSTD_BIN` | unset (search `PATH` for `zstd`) | `-zst` artifact refresh | Explicit zstd binary for unpacking `-zst` artifacts on Python < 3.14 (3.14+ uses the stdlib `compression.zstd` module). An absolute path for bundled or air-gapped deployments, or a bare name resolved against `PATH`. A value that resolves to nothing leaves `zst` refreshes refused with both remedies named; `xz` always works. |
  774. ### Pre-built database downloads
  775. | Variable | Default | Used by | Description |
  776. | :-------------------------- | :------------------------------------------ | :------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  777. | `VDB_APP_ONLY_DATABASE_URL` | `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` | `vdb db refresh full --app-only`, MCP auto-download | OCI image URL for the default app-only database. `vdb7-app-only` (a complete app-scope database), not `vdb7-app` (the partial group shard). Override to consume an internally published artifact. |
  778. | `VDB_DATABASE_URL` | `ghcr.io/appthreat/vdb7-full:v7.0.x-xz` | `vdb db refresh full`, `vdb db refresh full --image <oci-ref>` | OCI image URL for the app+OS database. Must be a v7 artifact; a v6 image is rejected. Prefer the `-xz` variant: it unpacks unconditionally, while `-zst` needs a zstd decompressor (Python 3.14+ or a `zstd` binary on PATH). |
  779. ### Source and feed selection for builds
  780. | Variable | Default | Used by | Description |
  781. | :----------------------------- | :------------------------------- | :------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  782. | `NVD_START_YEAR` | `2022` | NVD/vuln-list conversion, distro feeds | Start year for NVD-style CVE data and the Linux distro feeds. Older years increase coverage, build time and database size. Does not apply to application ecosystem advisories, because a distro fix moves the whole release forward while a lockfile does not. |
  783. | `VDB_APP_ECOSYSTEM_START_YEAR` | unset (no floor) | OSV application ecosystem feeds | Re-imposes a start year on the application ecosystems (npm, PyPI, Maven, Go, crates.io, NuGet, RubyGems, Hex, Packagist, Pub, CRAN, SwiftURL, Julia, GitHub Actions). A year, or `nvd` to follow `NVD_START_YEAR`. Set it only to trade coverage for a smaller artifact. |
  784. | `GITHUB_TOKEN` | unset | GitHub advisory ingestion | Token for the GitHub GraphQL API. Avoid printing this value in logs. |
  785. | `GITHUB_GRAPHQL_URL` | `https://api.github.com/graphql` | GitHub advisory ingestion | Alternate GitHub GraphQL endpoint, for testing or enterprise proxies. |
  786. | `GITHUB_PAGE_COUNT` | `2` | GitHub advisory ingestion | Number of GitHub advisory GraphQL pages to fetch during a full refresh. |
  787. | `NPM_PAGE_COUNT` | `2` | npm advisory configuration | Number of npm advisory pages to fetch where npm ingestion is used. |
  788. | `OSV_INCLUDE_FUZZ` | unset | OSV ingestion | Include the Linux, OSS-Fuzz and Android OSV feeds that are excluded by default to reduce false positives. |
  789. | `OSV_EXCLUDE_MALWARE` | unset | OSV conversion | Exclude OSV malware advisories whose identifiers start with `MAL`. |
  790. | `VDB_OSV_STORE_BATCH_SIZE` | `100` | OSV ingestion | Converted OSV records stored per database batch. Invalid values fall back to `100`; minimum 1. |
  791. | `VDB_MAX_AFFECTED_PER_BLOB` | `32` | CVE-5 blob storage | Maximum `affected` entries merged into one source blob per CVE; further entries spill into more blobs so blob size and per-row hydration cost stay flat as CVE fan-out grows. `0` disables merging; invalid values fall back to `32`. |
  792. | `VDB_INCLUDE_METADATA` | unset | CLI, storage, search metadata indexes | Populate extended metadata tables. Equivalent to `--include-metadata`. |
  793. | `VDB_METADATA_*` | unset | `vdb.meta` creation | Adds custom build metadata to `vdb.meta`; the prefix is stripped and the key lowercased. `true`/`1` and `false`/`0` are stored as booleans. |
  794. ### OS distro filtering for builds
  795. | Variable | Default | Used by | Description |
  796. | :------------------------- | :------ | :-------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  797. | `VDB_IGNORE_OS` | unset | OSV source configuration | Skip the OSV operating-system feeds added by default. Use app-only workflows for the smallest app database. |
  798. | `VDB_IGNORE_<DISTRO>` | unset | vuln-list and selected OSV distro feeds | Exclude distro-specific data. Static OSV toggles are `VDB_IGNORE_ALMALINUX`, `VDB_IGNORE_ALPINE`, `VDB_IGNORE_REDHAT`, `VDB_IGNORE_DEBIAN`, `VDB_IGNORE_ROCKYLINUX`, `VDB_IGNORE_MAGEIA`, `VDB_IGNORE_ALPAQUITA`, `VDB_IGNORE_MINIMOS`, `VDB_IGNORE_UBUNTU`, `VDB_IGNORE_AZURE_LINUX` and `VDB_IGNORE_SEAL`; vuln-list also supports the distro keys in `LINUX_DISTRO_VULN_LIST_PATHS` (including `VDB_IGNORE_ORACLE` and `VDB_IGNORE_BOTTLEROCKET`). SUSE and openSUSE are opt-in instead. Debian, Ubuntu, Red Hat, SUSE, openSUSE and Azure Linux data comes from the OSV feeds only. |
  799. | `VDB_EXCLUDE_<DISTRO>` | unset | vuln-list filtering | Alias for excluding vuln-list distro paths. |
  800. | `VDB_INCLUDE_<DISTRO>` | unset | vuln-list filtering | Force-include distro paths from `LINUX_DISTRO_VULN_LIST_PATHS`, such as `VDB_INCLUDE_ALPINE=true`. `VDB_INCLUDE_SUSE=true` and `VDB_INCLUDE_OPENSUSE=true` are separate: they enable the OSV SUSE and openSUSE feeds, the only source of SUSE family data, which are off by default. They have the heaviest errata fan-out of any feed, so they stay opt-in for local builds; the published OS databases enable both. |
  801. | `VDB_IGNORE_LINUX_KERNEL` | `true` | Storage | Skip `pkg:generic/linux` records. The kernel feed dominates the database (around 130k rows) with commit-sha version ranges that semantic-version lookups cannot match. |
  802. | `VDB_INCLUDE_LINUX_KERNEL` | unset | Storage | Opt back in to storing Linux kernel CVEs. Overrides `VDB_IGNORE_LINUX_KERNEL`. |
  803. ### Output and SQLite tuning
  804. | Variable | Default | Used by | Description |
  805. | :------------------------ | :------- | :-------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  806. | `VDB_QUIET` | unset | CLI | Suppress logo, logs and cache progress output. Equivalent to `--quiet`. |
  807. | `VDB_PROGRESS_INTERVAL` | `10000` | Source progress callbacks | Minimum records between progress messages. Invalid values fall back to `10000`; minimum 1. |
  808. | `VDB_SQLITE_IMMUTABLE` | unset | Search database connections | Open file-backed search databases with SQLite's immutable URI option, for read-only deployments where `.vdb7` files are not modified while the process runs. |
  809. | `VDB_SQLITE_CACHE_SIZE` | `-65536` | SQLite setup | Value passed to `PRAGMA cache_size`. The default is about 64 MiB using SQLite's negative-KiB convention. |
  810. | `VDB_SQLITE_JOURNAL_MODE` | `DELETE` | SQLite setup | Value passed to `PRAGMA journal_mode`. Valid: `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, `WAL`, `OFF`. |
  811. | `VDB_SQLITE_SYNCHRONOUS` | `OFF` | SQLite setup | Value passed to `PRAGMA synchronous`. Valid: `OFF`, `NORMAL`, `FULL`, `EXTRA`, or `0` to `3`. |
  812. | `VDB_DATA_PAGE_SIZE` | `32768` | Builds and splits | Page size for the data database. Powers of two from `512` to `65536`; invalid values warn and fall back. The 32K default ships around 27% smaller files and halves cold hydration; set `4096` to restore the previous size. |
  813. | `VDB_INDEX_PAGE_SIZE` | unset | Builds and splits | Page size for the index database. The random-access index file shows no win past the `4096` default, so leaving this unset is the recommendation. |
  814. | `PYTHONIOENCODING` | unset | Windows startup | If unset on Windows, VDB reconfigures standard streams to UTF-8. |
  815. ### MCP server
  816. | Variable | Default | Used by | Description |
  817. | :------------- | :------ | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
  818. | `VDB_AGE_DAYS` | `2` | MCP server | Days before the MCP server considers the local database stale and attempts an app-only ORAS download. Non-numeric values are passed through to the freshness check, so prefer an integer string. |
  819. ## Troubleshooting
  820. **Database locked, or `apsw.BusyError`.** Check that you are not running two
  821. `vdb --cache` processes at once. In a multi-threaded application, treat
  822. connections as read-only wherever possible. Reading one database while a build
  823. writes another can also raise this.
  824. **Disk space during a build or download.** The extracted full database is
  825. around 15.6 GB, and SQLite needs significant temporary space for index creation
  826. and `VACUUM` during a build. Point `VDB_TEMP_DIR` at a partition with room:
  827. ```shell
  828. export VDB_TEMP_DIR=/mnt/large_volume/vdb_temp
  829. vdb --cache-os
  830. ```
  831. **ORAS download failures.** Install the extra with
  832. `pip install appthreat-vulnerability-db[oras]`. Firewalls may block `ghcr.io`,
  833. in which case download manually. If your environment restricts outbound access
  834. or you need stronger provenance, publish databases from your own network and
  835. point clients at those artifacts.
  836. **A download that succeeds but leaves no database.** Check the compression
  837. suffix. `xz` unpacks unconditionally; a `-zst` URL unpacks only when a zstd
  838. decompressor is available (Python 3.14+ or a `zstd` binary on PATH) — the
  839. refresh refuses a zst fetch without one before downloading anything, naming
  840. both remedies.
  841. **`UnicodeEncodeError` on Windows.** VDB forces UTF-8 for stdout and stderr, but
  842. the terminal also has to be configured for it (`chcp 65001`).
  843. ## Model Context Protocol (MCP) server
  844. See the [MCP server readme](./packages/mcp-server-vdb/README.md).
  845. The server exposes structured tool results, resource templates such as
  846. `cve://{id}` and `purl://{purl}`, concrete resources such as `vdb://metadata`
  847. and `vdb://health`, bulk SBOM tools, prompts, and filter-aware search options
  848. covering severity thresholds, source filters, malware flags, package scope and
  849. pagination.
  850. ## Reading `.vdb7` files in other languages
  851. `.vdb7` files are standard SQLite databases; any modern SQLite library can read
  852. them. The tables, indexes and every query vdb itself runs are documented in
  853. [docs/DATABASE.md](./docs/DATABASE.md), and the design of the shard store,
  854. search fan-out and on-demand downloads in [docs/DESIGN.md](./docs/DESIGN.md).
  855. For agents and scripts doing aggregation or bulk export, querying directly is
  856. reasonable:
  857. ```bash
  858. sqlite3 "file:$VDB_HOME/data.index.vdb7?mode=ro" \
  859. "SELECT * FROM cve_index WHERE purl_prefix LIKE 'pkg:npm/react%';"
  860. ```
  861. Note that direct SQL sees none of the coverage, staleness or empty-database
  862. guards the library applies. On a shard, a purl of an uncovered type returns no
  863. rows, which is indistinguishable in SQL from clean. Prefer the library APIs
  864. when the answer will be reported to someone.
  865. ## For AI agents
  866. [SKILL.md](./SKILL.md) is the guide for using `vdb`, and
  867. [AGENTS.md](./AGENTS.md) is the guide for changing it. Both are written for
  868. agents. The essentials:
  869. Download a database before searching. `vdb db refresh full --app-only` for
  870. application packages, `vdb db refresh full` for OS packages. The refresh is
  871. staged and validated before it replaces the local database. Prefer a URL from
  872. the user's own workflow or mirror over the AppThreat-hosted default.
  873. Use the Python API for structured output. The CLI prints human-readable tables
  874. and has no `--json` flag on the search commands.
  875. Treat an empty result as a question, not an answer. Check that the search
  876. exited 0, that every requested purl type was covered, and whether any answering
  877. shard was stale. All three are reported.
  878. ## Local development
  879. Set up [uv](https://docs.astral.sh/uv/), then:
  880. ```shell
  881. uv sync --all-extras --dev
  882. uv run vdb --help
  883. uv run pytest
  884. uv run ruff check vdb test contrib
  885. ```
  886. To compare search latency while changing schemas, indexes or pragmas:
  887. ```shell
  888. uv run python contrib/bench_search.py \
  889. --data vdb_data/data.vdb7 \
  890. --index vdb_data/data.index.vdb7 \
  891. --query 'pkg:pypi/requests@2.31.0' \
  892. --repeat 5
  893. ```
  894. Pass `--with-data` to include CVE source-data decoding, or `--json` for
  895. machine-readable output.
  896. ## Citation
  897. ```text
  898. @misc{vdb,
  899. author = {Team AppThreat},
  900. month = Feb,
  901. title = {{AppThreat vulnerability-db}},
  902. howpublished = {{https://huggingface.co/datasets/AppThreat/vdb}},
  903. year = {2025}
  904. }
  905. ```
  906. ## License
  907. MIT

Ratings and comments

Sign in to rate this skill.

Sign in

Free to use. If it’s helped, pass it on.