Back to search
https://github.com/AppThreat/vulnerability-db
SkillWorks with Claude CodeGitHubIncomplete
Safety scan
A static text scan, not a guarantee. Select the badge to see what was scanned and every finding.
Checked today
Badge for your README
Markdown
[](/listings/appthreat--vulnerability-db)HTML
<a href="/listings/appthreat--vulnerability-db"><img src="/api/listings/appthreat--vulnerability-db/badge.svg" alt="findsafeskills safety rating" /></a>Install
i
Install instructions from the README
As written by the repo's author. findsafeskills has not verified these steps.
pip install appthreat-vulnerability-db>=7.0.0
For optional dependencies such as `oras`, install the `[oras]` or `[all]` group:
pip install appthreat-vulnerability-db[all]
Users of dep-scan v5 need the 5.8.x line instead:
pip install appthreat-vulnerability-db==5.8.0
What was scanned
The files the safety scan read, as written by the repository's author and shown as plain text. Lines with a finding are marked.
SKILL.md · Manifest · 348 lines
- # SKILL.md
- A practical guide for AI agents that need to use `vdb`. If you are changing the
- code rather than using it, read [AGENTS.md](./AGENTS.md) instead.
- Coming from VDB 6.x, read [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) before
- acting on any 6.x instructions you find elsewhere. Image URLs, environment
- variables and the comparison API changed, v7 refuses to read v6 databases, and
- 6.x code samples contain one API call that no longer exists.
- ## The rule that matters most
- An empty result is not a clean result until you have checked that the database
- exists, covers the purl types you asked about, and is fresh enough to matter.
- VDB reports all three, and the reports exist because a confident empty answer
- is the most dangerous output a vulnerability scanner can produce.
- Concretely, before you tell a user "no vulnerabilities found":
- ```mermaid
- flowchart TD
- A[Search returned no findings] --> B{Exit code 0?}
- B -- "no, exit 1 + 'empty'" --> X[Scanner down.<br/>Report a failure, not a clean scan.]
- B -- yes --> C{Shard store?}
- C -- "no, full database" --> D{Metadata search?}
- C -- yes --> E{Every requested purl<br/>type covered?}
- E -- "no, coverage_gap" --> Y[NOT CHECKED.<br/>Name the uncovered types.]
- E -- yes --> F{Any answering<br/>shard stale?}
- F -- yes --> Z[Answer may be old.<br/>Name the stale shard.]
- F -- no --> D
- D -- "yes, on a database<br/>without metadata" --> W[Tells you nothing.<br/>Build with --include-metadata.]
- D -- no --> OK[Clean result.<br/>Safe to report.]
- ```
- Only the `OK` branch is a clean result. Every other leaf is a statement about
- the database rather than about the software being scanned.
- ## Skill: choose the right database
- v7 publishes `vdb7-full`, `vdb7-app-only`, five app-scope variants, and
- per-type shards.
- | Need | Artifact |
- | :------------------------------------------------ | :------------------------------------------------------- |
- | Standard application dependency scanning | `vdb7-app-only` |
- | Container or Linux OS package scanning | `vdb7-full` |
- | CVE ID, alias, CPE, reference or full-text search | `vdb7-full` only; these raise on a shard |
- | Single-ecosystem workload (npm only, maven only) | the matching type shard |
- | Metadata, text, alias, reference or symbol search | `vdb7-app-extended` or `vdb7-app-10y-extended` |
- | Advisories older than 2020 | `vdb7-app-10y` / `vdb7-app-10y-extended` (NVD from 2016) |
- | Smallest download, recent advisories (CI) | `vdb7-app-2y` / `vdb7-app-2y-extended` (NVD from 2024) |
- `vdb7-app-only` is the app-scope _database_. `vdb7-app` is a different
- artifact: the app group _shard_, `completeness=partial`, for the shard store
- only. `vdb db refresh full` rejects it by design.
- Five shards are structural (`deb`, `rpm`, `apk`, `app`, and `cpe` as the
- complement) and the rest are per-type. There are 43 possible shard names and a
- build emits only the ones it has rows for, which was 26 on a recent app+OS
- build. Do not assume a fixed shard list; a shard's `vdb.meta`
- (`artifact.types`, `siblings.available`) is the authority, and `vdb db status`
- reports what is actually local.
- Prefer an organization-controlled mirror or an internally produced artifact
- when the user has one. Use the AppThreat-hosted defaults for bootstrap, local
- testing, or when no internal URL was given.
- Default v7 databases keep `cve_metadata` and `cve_metadata_text` empty to
- hold the index size down; the extended flavors (`vdb7-app-extended`,
- `vdb7-app-2y-extended`, `vdb7-app-10y-extended`) are the pre-built exceptions. When a task needs
- `search_full_text()`, `search_by_alias()`, `search_by_reference()`,
- `search_by_package_name()`, `search_by_symbol()`, or a metadata-dependent
- filter (source, severity, dates, malware flags), download the extended flavor
- with `vdb db refresh full --flavor app-extended`, build locally with
- `--include-metadata`, or point `VDB_DATABASE_URL` at an internal metadata
- artifact. On a default database those searches fail safe by returning no
- metadata matches, so an empty result there tells you nothing about the package.
- ## Skill: download before searching
- The files are `data.vdb7` and `data.index.vdb7` under `VDB_HOME`. v7 does not
- read or migrate `.vdb6`, and a search against a zero-row database exits 1 with
- an empty-database message rather than printing "No results found!".
- ```bash
- vdb db refresh full --app-only # app-only
- vdb db refresh full # app+OS
- vdb db refresh full --flavor app-extended # app-only + metadata tables
- ```
- Both stage and validate the download before it replaces the local database.
- For type shards:
- ```bash
- vdb db refresh npm pypi # named shards into the store
- ```
- The `-xz` image suffix is the safe choice: it always unpacks. A `-zst`
- reference works only when a zstd decompressor is present (Python 3.14+, or a
- `zstd` binary on PATH); without one the refresh fails before downloading and
- names both remedies. `vdb db refresh --compression zst` selects zst
- explicitly.
- In Python:
- ```python
- import os
- from vdb.lib import config, db, search
- from vdb.lib.orasclient import download_image
- DB_URL = os.getenv("VDB_APP_ONLY_DATABASE_URL", config.VDB_APP_ONLY_DATABASE_URL)
- if db.needs_update(days=1, default_status=True):
- download_image(DB_URL, config.DATA_DIR)
- results = search.search_by_any("pkg:pypi/requests@2.31.0", with_data=True)
- ```
- Set `VDB_HOME` before importing `vdb.lib.config`. Path and build-scope
- variables are read at import time.
- ## Skill: get structured results
- The CLI prints human-readable tables. There is no `--json` flag on the search
- commands, so do not look for one. Use the Python API for machine-readable
- output.
- ```python
- import json
- from vdb.lib import search
- output = []
- for item in search.search_by_any("pkg:npm/lodash@4.17.20", with_data=True):
- row = {
- "cve_id": item["cve_id"],
- "package": item["name"],
- "purl_prefix": item["purl_prefix"],
- "vers": item["vers"],
- "fix_version": item.get("fix_version"),
- "severity": item.get("severity"),
- }
- if item.get("source_data"):
- row["cve_data"] = item["source_data"]
- output.append(row)
- print(json.dumps(output, indent=2))
- ```
- `source_data` is a plain dict, ready for `json.dumps()`. It is not a Pydantic
- model and has no `.model_dump()`. Older snippets call
- `item["source_data"].model_dump(mode="json")`; that raises `AttributeError` on
- v7. This is the single most common wrong line in 6.x-era sample code.
- ## Skill: bulk searches
- ```python
- from vdb.lib import search
- packages = [
- {"purl": "pkg:pypi/requests@2.31.0"},
- {"purl": "pkg:npm/react@18.2.0"},
- {"url": "https://github.com/pallets/flask"},
- ]
- for batch in search.search_packages_batched(packages, batch_size=100, with_data=False):
- for item in batch:
- print(item["locator"], item["result_count"], item.get("max_severity"))
- ```
- For a CycloneDX BOM, `search_bom_summary` and `search_bom_detailed` take the
- same filters, and `search_by_cdx_bom` yields batches so peak memory stays
- bounded on a large SBOM.
- ## Skill: manage local data with `vdb db`
- ```bash
- vdb db status # main DB and per-shard build id, age, size, coverage
- vdb db status --json # the same report, machine-readable
- vdb db refresh # sync what I have: everything already in the store
- vdb db refresh npm pypi # a named set of shards (names or purl types)
- vdb db refresh --all # every default shard
- vdb db refresh full # the full image, staged and validated
- ```
- Call `vdb db status` before reporting scan results, and again after any refresh.
- The plain and JSON reports expose the same facts: per-shard `build_id`, build
- time and `age_days`, sizes, `mixed_builds`, `missing_group_shards`, `coverage`
- (`full`, or `partial (missing: ...)`), and a per-shard `stale` flag.
- Refreshes are safe to interrupt. The full image is downloaded to a staging
- directory, validated (both `.vdb7` files present, index non-empty, not a
- single-shard artifact) and swapped in only after validation, with the current
- files stashed away first. Each shard lands with an atomic rename. A failed or
- empty fetch is an error and never replaces a working database.
- ## Skill: read coverage and staleness honestly
- When the connected main database is a shard (`vdb.meta` says
- `completeness: "partial"`), purl searches fan out across the local shard store.
- Two conditions that look similar and are not:
- **Coverage gap.** A requested purl type has no local shard. Single-purl
- searches raise `ShardCoverageError`. Batch searches mark the affected
- components `coverage_gap: true`, expose requested, covered and uncovered types
- through the `.coverage` companion on the result list and
- `vdb.lib.search.last_shard_coverage`, and warn on stderr.
- **Staleness.** A shard more than 7 days older than its newest sibling. Reported
- per shard by `vdb db status` and warned at search time. A shard with no
- parseable build time is never claimed stale, though its `build_id` is still
- shown.
- Rules for an agent:
- An empty result on a shard store is not "no vulnerabilities" until `coverage`
- shows every requested type was covered. A `coverage_gap` means not checked.
- A stale shard still answers, and the answer may be old. Name the stale shard
- when you report results.
- Never conclude "clean" from a scan whose `vdb db status` shows
- `missing_group_shards`.
- Never enable `VDB_AUTO_FETCH` on a user's behalf in air-gapped CI. On-demand
- network fetch during a scan is opt-in for a reason.
- ## Skill: build a local database safely
- Use app-only builds unless OS package scanning is required.
- ```bash
- export VDB_HOME=/tmp/vdb-home
- export VDB_CACHE=/tmp/vdb-cache
- export VDB_TEMP_DIR=/tmp/vdb-home/tmp
- export NVD_START_YEAR=2024
- export GITHUB_PAGE_COUNT=1
- mkdir -p "$VDB_HOME" "$VDB_CACHE" "$VDB_TEMP_DIR"
- vdb --cache
- ```
- `NVD_START_YEAR` bounds NVD-style CVE data and the Linux distro feeds only.
- Application ecosystem advisories are not year-filtered, because a distro fix
- moves the whole release forward while a lockfile does not, so an old library
- advisory is exactly what a scan needs. Set
- `VDB_APP_ECOSYSTEM_START_YEAR` if you specifically want the smaller artifact.
- Do not set `OSV_EXCLUDE_MALWARE=true` to save space on a build you will scan
- with. It removes the malicious-package advisories, and a missed malicious
- package is an installed backdoor rather than a stale finding.
- For app+OS builds, and for metadata search support:
- ```bash
- vdb --cache-os
- vdb --cache --include-metadata
- vdb --cache-os --include-metadata
- ```
- Builds print minimal progress. Use `--quiet` or `VDB_QUIET=true` to suppress
- the logo, logs and progress. Tune cadence with `VDB_PROGRESS_INTERVAL`.
- To restrict OS data, use the `VDB_IGNORE_*` and `VDB_INCLUDE_*` flags from
- `vdb/lib/config.py`:
- ```bash
- export VDB_IGNORE_DEBIAN=true
- export VDB_IGNORE_ALPINE=true
- vdb --cache-os
- ```
- Set `VDB_TEMP_DIR` to a partition with room. An app+OS build needs significant
- temporary space for index creation and `VACUUM`, and a small `/tmp` is the most
- common build failure.
- ## Skill: reproduce cache bugs without large downloads
- `AquaSource` reads `$VDB_CACHE/vuln-list.zip` when it exists. Build a small zip
- with a few representative paths and monkeypatch `config.CACHE_DIR` at it in
- tests. This avoids cloning or downloading the full vuln-list repository.
- ## Skill: reason about v7 storage
- `data.vdb7` stores each CVE source blob once in `cve_source_data`, with package
- locator rows in `cve_data` referencing them by hash. `data.index.vdb7` stores
- package and version rows in `cve_index`, plus optional metadata in
- `cve_metadata` (compact, package-level) and `cve_metadata_text` (long text, one
- row per CVE).
- The split follows access patterns: the index is small, random-access and hot;
- the data file is large, sequential and cold. Resolve against the index first,
- hydrate only what matched.
- Databases are built from scratch by the release workflows. Schema migrations
- and legacy inline `cve_data.source_data` write paths are deliberately not
- required. v7 reads `.vdb7` only.
- ## Skill: direct SQLite inspection
- The `.vdb7` files are ordinary SQLite databases.
- ```bash
- sqlite3 "$VDB_HOME/data.index.vdb7" ".schema cve_index"
- sqlite3 "$VDB_HOME/data.index.vdb7" "SELECT count(*) FROM cve_index;"
- sqlite3 "$VDB_HOME/data.index.vdb7" "SELECT count(*) FROM cve_metadata;"
- sqlite3 "$VDB_HOME/data.vdb7" "SELECT count(*) FROM cve_data;"
- sqlite3 "$VDB_HOME/data.vdb7" "SELECT count(*) FROM cve_source_data;"
- ```
- A healthy build has many more `cve_data` rows than `cve_source_data` rows,
- because a CVE affecting many packages shares one blob. In a default database
- `cve_metadata` and `cve_metadata_text` have zero rows; in an
- `--include-metadata` build they are populated.
- Open a database you did not build read-only:
- ```bash
- sqlite3 "file:$VDB_HOME/data.index.vdb7?mode=ro" "SELECT count(*) FROM cve_index;"
- ```
- Two traps when querying by hand. Locators are lowercased, so lowercase your
- query. And a bare `SELECT ... FROM cve_index` scans in name order via a
- covering index rather than by CVE id, so ordering by `cve_id` costs nothing but
- assuming an order costs correctness.
- Direct SQL sees none of the coverage, staleness or empty-database guards the
- library applies. On a shard, a purl of an uncovered type returns no rows, and
- that is indistinguishable in SQL from clean. Prefer the library APIs when the
- answer will be reported to someone.
- ## Skill: investigate a false positive or false negative
- Trace in order:
- 1. The source converter (`osv.py`, `aqua.py`, `gha.py`, `nvd.py`), including
- whether the record survived its feed's year floor at all.
- 2. `VulnerabilityDetail` fields: `mii`, `mie`, `mai`, `mae`, `fixed_location`,
- `package_type`.
- 3. `to_purl_vers()` in `utils.py`.
- 4. `CVESource.store5()`, for the purl prefix and index row it writes.
- 5. The search path and `vdb.lib.vers.vers_compare()`.
- For a false negative, check step 4 against step 5 specifically. The most common
- cause is a stored locator the lookup never proposes: both sides build a purl
- prefix independently, and when they disagree the rows exist and are
- unreachable, which looks exactly like "not vulnerable".
- Add the regression test at the lowest layer that demonstrates the bug.
README.md · README · 1,121 lines
- # appthreat-vulnerability-db
- An offline vulnerability database and package search library. `vdb` downloads
- data from AppThreat vuln-list, OSV, NVD and GitHub, converts it into the CVE 5.2
- schema, and stores it in SQLite with indexes built for fast offline lookup by
- purl, CPE or git URL.
- Every row uses an open specification, CVE 5.2 for the records and Package URL
- (purl and vers) for the locators, so the data stays readable without this
- library. The `.vdb7` files are ordinary SQLite databases.
- Upgrading from 6.x? Read [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) first. The
- database files, the published image URLs, three environment variables and the
- version comparison API all changed, and v7 does not read v6 databases.
- Measured v6-versus-v7 figures for build time, database size, query latency and
- shard footprint are in [docs/BENCHMARKS.md](./docs/BENCHMARKS.md).
- The full documentation, including a CLI reference, an environment variable
- index and hands-on tutorials, is published at
- [appthreat.github.io/vulnerability-db](https://appthreat.github.io/vulnerability-db/)
- from the [docs](./docs) directory.
- ## What it is for
- Advisories are merged from several upstream feeds because the same CVE is
- often published independently by NVD, an OSV feed and a GitHub advisory, with
- different affected ranges; querying the union of them reduces false negatives.
- Purl prefixes are generated even for git repositories, so a search can start
- from a purl, a CPE, or an HTTP git URL.
- Lookups run entirely against local SQLite files; there is no network access
- during a scan. Missing data is reported explicitly instead of silently: an
- absent or emptied database exits 1 with an error message, a purl type that no
- local shard serves raises `ShardCoverageError`, and batch searches mark
- uncovered components `coverage_gap` rather than returning an empty list.
- ```
- vdb --cache / --cache-os
- Upstream feeds -------------------------> data.vdb7 (CVE 5.2 records)
- data.index.vdb7 (purl/vers index)
- |
- optional split
- v
- shards/<type>/ (per-type databases)
- ```
- ## Data sources
- Linux [vuln-list](https://github.com/appthreat/vuln-list) (forked from Aqua
- Security), OSV, NVD and GitHub.
- Linux and OSS-Fuzz OSV feeds are excluded by default; set `OSV_INCLUDE_FUZZ=true`
- to include them. Malware feeds are included by default and increase the database
- size slightly; `OSV_EXCLUDE_MALWARE=true` excludes them, though a missed
- malicious package is a worse outcome than a larger artifact.
- Application ecosystem advisories are not filtered by year. `NVD_START_YEAR`
- bounds NVD-style CVE data and the Linux distro feeds only. The reason is that a
- distro fix moves the whole release forward, so an old distro advisory describes
- versions nobody runs, whereas a lockfile does not move, and `lodash@4.17.15` or
- `log4j-core@2.14.1` are in projects today.
- ### Linux distros
- AlmaLinux, Alpine, Amazon Linux, Arch Linux, Azure Linux (CBL-Mariner),
- Bottlerocket, Debian, Oracle Linux, RHEL/CentOS, Rocky Linux, Ubuntu, Photon,
- Chainguard, Wolfi OS, Mageia, Alpaquita, MinimOS, and SLES plus openSUSE Leap
- and Tumbleweed as opt-in feeds.
- A few details that affect how rows are stored and matched:
- Debian, Ubuntu, Red Hat, SUSE and openSUSE come from the corresponding OSV
- feeds, which carry per-release fixed versions and CVE ids. Red Hat and SUSE
- errata fan out to every CVE they name. SUSE family rows are stored under
- channel-scoped `pkg:rpm/suse|opensuse/<release>/<name>` locators that match the
- `distro` qualifier cdxgen emits. The equivalent vuln-list subtrees duplicated
- this data less precisely, or under locators no scanner emits, and are ignored at
- ingest.
- Azure Linux comes from the OSV `Azure Linux` feed, which publishes `AZL-*`
- advisories carrying one CVE each with per-release fixed versions. Rows use
- channel-scoped `pkg:rpm/azure-linux/<release>/<name>` locators, and the
- `azurelinux`, `cbl-mariner` and `mariner` purl namespaces cdxgen emits all alias
- onto `azure-linux`.
- Photon rows are release-scoped as `pkg:rpm/photon/photon-<release>/<name>`.
- Bottlerocket comes from the vuln-list `bottlerocket` tree (BRSA and
- GHSA-id'd updateinfo advisories). Rows are flat `pkg:rpm/bottlerocket/<name>`
- locators — there is no bottlerocket purl type in the spec, and cdxgen
- namespaces rpms by the os-release ID; a trivy-style `pkg:bottlerocket/<name>`
- scan purl is folded onto the same locator — with full version-release bounds
- and the CVE taken from the advisory references. Package entries whose release
- is a bare commit hash name the Bottlerocket OS release that shipped the fix,
- not a package version, and are skipped. `VDB_IGNORE_BOTTLEROCKET=true`
- excludes the tree.
- Oracle Linux comes from the vuln-list `oval/oracle` tree (ELSA errata, and
- ELBA errata that name CVEs). Rows are scoped by major release,
- `pkg:rpm/oracle/oracle-<N>/<name>` (module streams
- `oracle-<N>-stream<S>`): Oracle ships every fix in the latest repository, so
- a host on an older minor release still matches. The release comes from each
- bound's `el` tag (including `elNuek`) or from the `Oracle Linux N is
- installed` criterion of its own branch, Ksplice-only branches are skipped, and
- `distro=oracle-8.10`, `distro=ol-8.10` and the `pkg:rpm/ol/...` namespace all
- fold onto the same channel. Each row carries its own CVE's CVSS3 vector and
- metrics from the erratum. `VDB_IGNORE_ORACLE=true` excludes the tree.
- ### Seal Security
- The two `vulnfeed.sealsecurity.io` OSV feeds carry CVE advisories for the
- packages Seal rebuilt with backported fixes, under Seal's own names
- (`seal-requests`, `@seal-security/ajv`) and under the original names with
- Seal version markers (`requests@2.14.2+sp1`). Every stored range must sort
- above the upstream release it rebuilds under the ecosystem's own comparator;
- ranges that do not are dropped rather than widened, so upstream releases never
- match. Seal:PyPI ranges are kept, because the pypi comparator follows PEP 440 and
- sorts a local version such as `1.26.18+sp0` above the release `1.26.18`. A few
- Go `+incompatible` and Maven ranges are dropped. Seal's `+sp999` ceiling is reported as no
- fix. The feeds are registered in app-only builds too; those keep the
- application ecosystems and skip Seal's Debian, Ubuntu, Alpine and Red Hat
- rows. `VDB_IGNORE_SEAL=true` excludes both feeds.
- Chainguard, Wolfi and MinimOS advisories filed under a fix version of `0`
- are not stored. That is the Alpine secdb and OSV spelling of "not affected"
- (a false-positive determination), not "no fix yet".
- ## Installation
- ```shell
- pip install appthreat-vulnerability-db>=7.0.0
- ```
- For optional dependencies such as `oras`, install the `[oras]` or `[all]` group:
- ```shell
- pip install appthreat-vulnerability-db[all]
- ```
- Users of dep-scan v5 need the 5.8.x line instead:
- ```shell
- pip install appthreat-vulnerability-db==5.8.0
- ```
- ## Getting a database
- This package is primarily a library, used by
- [owasp-dep-scan](http://github.com/owasp-dep-scan/dep-scan). The CLI exists to
- test and explore it directly.
- > [!IMPORTANT]
- > The AppThreat-hosted database images and workflows are best treated as
- > bootstrap or evaluation defaults. For production, especially for the larger
- > app+OS variants, build and publish your own databases from your own CI.
- >
- > You control when data is built, where it is published, and which upstream
- > sources and retention windows are allowed. You can publish smaller artifacts
- > that match your environment instead of pulling a one-size-fits-all image. And
- > app+OS builds need significant compute, disk and network, which is easier to
- > budget for on infrastructure you own.
- ### Option 1: `vdb db` for staged, validated refreshes (preferred)
- `vdb db` is the front end for getting and refreshing data: the full image, a
- named set of type shards, or everything already present locally. It is the
- preferred way to obtain a database, because every refresh is staged, validated
- and swapped in atomically.
- ```shell
- vdb db status # what is local: main DB and per-shard build id, age, size, coverage
- vdb db refresh full # the full image, staged, validated and swapped
- vdb db refresh full --flavor app-extended # app-scope + metadata tables
- vdb db refresh npm pypi # named shards (shard names or purl types; nuget resolves to app)
- vdb db refresh --all # every shard the local build published (siblings.available)
- vdb db refresh # sync what I have: everything already in the local store
- vdb db refresh npm --tag v7-xz # a different release line (see below)
- ```
- Shard tags always carry an encoding suffix, because that is all the build
- publishes: `vdb7-npm:v7.0.x-xz` and `vdb7-npm:v7.0.x-zst` exist, a bare
- `vdb7-npm:v7.0.x` does not. `vdb db refresh` appends it for you, inheriting
- whichever encoding the local store was fetched with and defaulting to `xz`.
- `zst` works when a zstd decompressor is available — Python 3.14+ (which ships
- `compression.zstd`) or a `zstd` binary on PATH (`VDB_ZSTD_BIN` points at one
- elsewhere); without one, a zst refresh
- fails before downloading anything and says so. `xz` needs no decompressor,
- which is why it stays the default. Use `--tag` to pick a release line rather
- than to restore a suffix.
- The remaining refresh flags: `--registry` points shards at your own registry
- (default `ghcr.io/appthreat`, or the siblings record of a local shard),
- `--store-dir` overrides the shard store location (default `$VDB_SHARDS_DIR`),
- and `--compression xz` makes the encoding choice explicit. `refresh full`
- also accepts `--flavor <name>` to pick one of the published full-database
- variants (`full`, `app-only`, `app-extended`, `app-2y`, `app-2y-extended`,
- `app-10y`, `app-10y-extended`;
- see [Scope variants](#scope-variants)), `--app-only` as the shorthand for
- `--flavor app-only`, and `--image <oci-ref>` for a single-run URL override
- that wins over both.
- `vdb db status` reports, per shard, the build id, build time and age, sizes,
- type coverage, whether the store is full or partial (naming any missing group
- shards), whether it mixes builds, and which shards are much older than their
- siblings. Mixing builds is a normal state, because shards are independent
- artifacts. `--json` emits the same report machine-readably.
- Refreshes are safe to interrupt. Shard refreshes place each shard with an atomic
- directory rename. The full-image refresh downloads into a staging directory,
- validates the artifact (both `.vdb7` files present, index non-empty, not a
- single-shard artifact), and only then swaps it in, stashing the current files
- away before the new ones land. An interrupted refresh therefore leaves the main
- directory with files _missing_, which is loudly reported as an empty database
- with exit 1, rather than a mixed old and new pair that could answer subtly
- wrong. A failed or empty fetch is an error and never replaces a working
- database.
- If you use dep-scan: `vdb db` subsumes the mechanism `depscan-vdb` provides,
- which is downloading an image into `VDB_HOME`, fetching and refreshing shards,
- and reporting local state. Variant selection, retry policy and the freshness
- marker remain dep-scan's concerns. See
- [MIGRATING_TO_V7.md](./MIGRATING_TO_V7.md) section 3.
- ### Option 2: download a pre-built database
- The app-only database (about 160 MB to download, around 1 GB on disk) is
- [refreshed](https://github.com/AppThreat/vdb/actions) every 12 hours and is the
- fastest way to evaluate `vdb`, bootstrap a workstation, or validate an
- integration.
- ```shell
- vdb db refresh full --app-only
- ```
- For container and OS scans, download the full image (about 730 MB to download,
- 15.6 GB on disk):
- ```shell
- vdb db refresh full
- ```
- To consume an artifact your own workflow published, override the URL:
- ```shell
- export VDB_DATABASE_URL=your.registry/vdb7-full:your-tag-xz
- vdb db refresh full
- ```
- Keep the `-xz` suffix on any URL you set: it unpacks with no further
- requirements. A `-zst` URL also works, but needs a zstd decompressor
- (Python 3.14+, or a `zstd` binary on PATH); without one the download is
- refused up front with both remedies named.
- Metadata searches (full text, alias, reference, package name, symbol, source,
- severity, date and malware-aware filters) need a database whose metadata tables
- are populated. The v7 build publishes those as the extended flavors —
- app-scope, NVD from 2020 or 2016:
- ```shell
- vdb db refresh full --flavor app-extended # NVD 2020+ + metadata tables
- ```
- Use `--flavor app-10y-extended` for metadata plus NVD back to 2016. For any
- other scope, build one yourself:
- ```shell
- vdb --cache --include-metadata
- ```
- ### Option 3: ORAS
- ```shell
- export VDB_HOME=$HOME/vdb
- oras pull ghcr.io/appthreat/vdb7-full:v7.0.x-xz -o $VDB_HOME
- tar -xvf *.tar.xz
- rm *.tar.xz
- ```
- Use `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` for the app-only database, or
- `ghcr.io/appthreat/vdb7-<shard>:v7.0.x-xz` for a single type shard. Note the
- `-only`: `vdb7-app` is the shard store's app group shard, a partial slice of
- the full database, not a database you install on its own.
- `vdb db refresh` does this for you and validates what it downloads. The `-zst`
- artifacts published alongside each `-xz` one are smaller to download; vdb can
- fetch them too when a zstd decompressor is present (Python 3.14+ or a `zstd`
- binary), and manual pipelines can always unpack zstd themselves.
- ### Option 4: Hugging Face
- The v7 artifacts live under `v7-<artifact>/` prefixes of the
- [AppThreat/vdb](https://huggingface.co/datasets/AppThreat/vdb) dataset:
- `v7-full/`, `v7-app/`, `v7-npm/` and so on.
- ```shell
- pip install -U "huggingface_hub[cli]"
- export VDB_HOME=$(pwd)/app
- hf download --repo-type dataset AppThreat/vdb v7-app/data.vdb7.zst --local-dir .
- ```
- Uncompressed `.vdb7` files are on Hugging Face only, and only for the artifacts
- small enough to be worth it. `full`, `deb`, `rpm` and `apk` are published
- compressed and are best fetched over ORAS with `vdb db refresh`.
- ### Option 5: build your own (recommended for production)
- If you depend on `vdb` regularly, build and publish your own databases
- internally. This is the recommended approach for enterprises, security teams
- and integrators: you publish from infrastructure you trust, you reduce
- supply-chain and availability dependencies on third-party refresh jobs, you can
- tune the scope to your environment, and you can put larger app+OS builds on
- runners sized for them.
- The workflow is to set your retention and distro selection variables, run
- `vdb --cache` or `vdb --cache-os` on a schedule, package the resulting `.vdb7`
- files, publish them to your own registry or object store, and point clients at
- your URL.
- ```shell
- vdb --cache # application vulnerabilities, around 1 GB
- vdb --cache --include-metadata # the same, with metadata search tables
- vdb --cache-os # application and OS vulnerabilities, around 15.6 GB
- vdb --cache-os --include-metadata
- vdb --cache --only-osv # OSV as the only source
- vdb --clean # remove existing databases
- ```
- `--include-metadata` is equivalent to `VDB_INCLUDE_METADATA=true`. Builds print
- minimal progress, such as the source year being fetched or the latest CVE
- stored; `--quiet` suppresses the logo, logs and progress.
- Two knobs change coverage rather than just size:
- ```shell
- NVD_START_YEAR # default 2022, supports back to 2002. NVD-style data and distro feeds only.
- GITHUB_PAGE_COUNT # default 2, supports up to 20
- ```
- A team that only scans modern application dependencies can build a much smaller
- artifact with a recent `NVD_START_YEAR`, noting that
- `VDB_APP_ECOSYSTEM_START_YEAR` is unset by default so library advisories are
- kept regardless of age. A platform team supporting a subset of distros can use
- the `VDB_IGNORE_*` or `VDB_INCLUDE_*` variables before `vdb --cache-os`:
- ```shell
- export VDB_IGNORE_ALMALINUX=true
- export VDB_IGNORE_UBUNTU=true
- vdb --cache-os
- ```
- See `LINUX_DISTRO_VULN_LIST_PATHS` in [config.py](./vdb/lib/config.py) for the
- full list of distro strings.
- Set `VDB_TEMP_DIR` to a partition with room. An app+OS build needs significant
- temporary space for index creation and `VACUUM`, and a small `/tmp` is the most
- common build failure.
- ## The two database files
- **`data.index.vdb7`** is the search side: small, random-access, hot. It holds
- `cve_index`, one row per `(cve_id, vers, purl_prefix)`, plus the optional
- `cve_metadata` and `cve_metadata_text` tables.
- **`data.vdb7`** is the payload side: large, sequential, cold. It holds
- `cve_source_data`, where each CVE 5.2 source blob is stored once and keyed by
- hash, and `cve_data`, the package locator rows that reference those blobs.
- The split follows the access patterns. You resolve a package against the index
- first, which is cheap, and hydrate only what matched. A CVE affecting 300
- packages costs one blob rather than 300 copies.
- Default public databases leave `cve_metadata` and `cve_metadata_text` empty to
- minimise index size, and v7 publishes no artifact with them populated. Build
- with `--include-metadata` for text, alias, reference, symbol, severity, source
- and date-aware searches, or publish your own metadata artifact and point
- `VDB_DATABASE_URL` at it. Public artifacts are rebuilt from scratch by the
- release workflows, so migrations for older `.vdb6` files are not maintained.
- Any SQLite browser or CLI can read both files. See
- [INTEGRATION.md](./INTEGRATION.md) for the schemas and query patterns.
- ## Type-shard databases
- The full, app-only and per-type shard images are all published and usable
- today: `ghcr.io/appthreat/vdb7-full:v7.0.x-xz`,
- `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` and
- `ghcr.io/appthreat/vdb7-<type>:v7.0.x-xz` for every shard the build produced,
- plus `-zst` variants and the floating `v7-*` and `latest-*` tags. A shard's own
- `vdb.meta` (`artifact.types` and `siblings.available`) is the authority on what
- exists, because the shard set grows when a new ecosystem first gets an
- advisory; check a specific artifact with
- `oras manifest fetch <ref> --descriptor`.
- A shard is a standalone database holding only the purl types it declares, so a
- scanner that only sees npm packages can download a fraction of the full
- artifact and get byte-identical answers for those types. On disk, a `pypi`-only
- workload needs about 90 MB of shard (82.7 MB data plus 7.4 MB index) instead
- of the 15.6 GB full production pair, a roughly 170-fold reduction in
- uncompressed bytes. Both figures are uncompressed on-disk sizes; the published
- images are xz- or zst-compressed and the compressed ratio differs, so compare
- download sizes against download sizes rather than mixing the two.
- Five shards are structural: `deb`, `rpm`, `apk`, `app` (the application
- ecosystem union) and `cpe`, which is the complement holding every purl-prefix
- type no other shard claims. The rest are per-type, one for each
- [purl-spec type](https://github.com/package-url/purl-spec/tree/main/types)
- except `deb`, `rpm`, `apk` and `generic`. That is 43 possible shard names, and
- the splitter emits only the ones the build actually has rows for, which was 27
- on the 2026-08-23 app+OS build. A shard's own `vdb.meta` (`artifact.types` and
- `siblings.available`) is the authority on what exists and what it serves, so do
- not hard-code a shard list. Purl types the spec does not define stay in `cpe`.
- Per-type shards deliberately overlap the group shards rather than partitioning
- them: the `npm` shard and the `app` shard both serve npm. They are separate
- views, which is why removing a type from `cpe` would strand it for a store that
- only carries the group shards.
- To use shards, point your download flow at a shard artifact, which becomes the
- connected main database under `VDB_HOME` with `completeness: "partial"` in its
- `vdb.meta`, or fetch shards into the store with `vdb db refresh`. Additional
- shards go under `$VDB_SHARDS_DIR` (default `$VDB_HOME/shards`).
- Purl searches (`search_by_purl_like`, `search_packages_batched`,
- `search_bom_*`) fan out across the local shards automatically. CVE, alias, CPE
- and text searches require the full database and raise on a shard rather than
- under-report.
- Shard-mode results carry a coverage companion (`.coverage` on the returned
- list, `vdb.lib.search.last_shard_coverage`, and a `coverage` key on BOM
- results) reporting which purl types were checked. Uncovered types are marked
- `coverage_gap` and warn on stderr, so a partial scan cannot look clean. A shard
- more than 7 days older than its siblings triggers a staleness warning at search
- time, and a _missing_ shard stays a distinct condition (a coverage gap, or
- `ShardCoverageError`) rather than being collapsed into staleness. Set
- `VDB_AUTO_FETCH=true` to let searches fetch missing shards on demand, or call
- `vdb.lib.shard_store.fetch_shards()` explicitly.
- See `contrib/shard_store.md` for the design, measurements and safety argument, and
- `contrib/shard_split.md` for the splitter.
- ### Automatic shard splitting
- After a successful `vdb --cache` or `vdb --cache-os` build, the database is
- split by purl type into the shards directory. Splitting runs only when the build
- completed and the result is complete (`completeness: full` in `vdb.meta`), so a
- partial or failed build never produces a shard set. Shards that would be empty
- for your configuration, such as the OS distro shards of an application-only
- build, are skipped. If the split fails, the failure is reported and the
- partially written shards are discarded, while the full database that was just
- built stays intact and usable.
- ```shell
- vdb --cache --no-split
- ```
- `--no-split` is honoured by every entry point that triggers a build: the `vdb`
- console script, `python -m vdb.cli`, and library calls to `vdb.cli.main()`.
- The splitter is also available for re-splitting an existing database or
- verifying a shard set. The source database is opened read-only.
- ```shell
- vdb split --db-file ~/.vdb/data.vdb7 --index-file ~/.vdb/data.index.vdb7 -o /path/to/shards -v
- vdb verify --db-file ~/.vdb/data.vdb7 --index-file ~/.vdb/data.index.vdb7 --shards-root /path/to/shards
- ```
- `vdb split` accepts `--shards`, `--build-id`, `--version-tag`, `--jobs`
- (`--jobs 0` uses one worker per CPU) and `--report` to write a split report
- JSON. `vdb verify` checks that the group shards exactly partition the full
- database and exits non-zero when they do not.
- ## Published artifacts and their sizes
- The shard set is not fixed: the splitter emits one `vdb7-<type>` artifact for
- every purl type the build actually has rows for, so it grows when a new
- ecosystem first gets an advisory (this build produced 27). The full artifact's
- `vdb.meta` (`siblings.available`) is the authority on what exists.
- If you are choosing automatically, pick the smallest row whose types cover the
- SBOM you are scanning. `full` is only needed for CPE lookups or when you cannot
- predict the types in advance.
- | Artifact | Purl types served | Rows | Uncompressed data | Uncompressed index | `.tar.xz` | `.zst` |
- | :----------------- | :------------------------------------------------------------------------------ | --------: | ----------------: | -----------------: | --------: | ------: |
- | `vdb7-full` | everything, including CPE | 6,894,338 | 12.2 GiB | 2.3 GiB | 733 MB | 806 MB |
- | `vdb7-deb` | `deb` | 2,995,928 | 4,139 MB | 928 MB | 153 MB | 166 MB |
- | `vdb7-rpm` | `rpm` | 2,084,872 | 6,171 MB | 977 MB | 235 MB | 271 MB |
- | `vdb7-apk` | `apk` | 1,137,398 | 1,294 MB | 359 MB | 104 MB | 111 MB |
- | `vdb7-app` | `npm`, `pypi`, `maven`, `golang`, `nuget`, `gem`, `composer`, `cargo`, `github` | 396,377 | 923 MB | 109 MB | 161 MB | 170 MB |
- | `vdb7-cpe` | the complement: 35 types no other shard claims, incl. `generic` | 279,763 | 552 MB | 89 MB | 48.8 MB | 51.7 MB |
- | `vdb7-npm` | `npm` | 237,664 | 509 MB | 56 MB | 50.0 MB | 52.7 MB |
- | `vdb7-github` | `github` | 63,122 | 161 MB | 23 MB | 20.3 MB | 21.6 MB |
- | `vdb7-pypi` | `pypi` | 31,823 | 82.7 MB | 7.4 MB | 8.2 MB | 8.8 MB |
- | `vdb7-maven` | `maven` | 20,225 | 43.0 MB | 7.7 MB | 4.1 MB | 4.5 MB |
- | `vdb7-composer` | `composer` | 17,178 | 44.1 MB | 5.7 MB | 4.3 MB | 4.7 MB |
- | `vdb7-golang` | `golang` | 8,458 | 45.4 MB | 3.3 MB | 4.8 MB | 5.3 MB |
- | `vdb7-nuget` | `nuget` | 8,040 | 20.5 MB | 3.1 MB | 1.1 MB | 1.2 MB |
- | `vdb7-gem` | `gem` | 6,138 | 12.4 MB | 1.4 MB | 1.2 MB | 1.3 MB |
- | `vdb7-cargo` | `cargo` | 3,729 | 12.4 MB | 1.1 MB | 1.4 MB | 1.5 MB |
- | `vdb7-julia` | `julia` | 2,040 | 5.7 MB | 0.6 MB | 610 KB | 635 KB |
- | `vdb7-alpm` | `alpm` | 1,500 | 2.0 MB | 0.4 MB | 175 KB | 185 KB |
- | `vdb7-hex` | `hex` | 293 | 1.3 MB | 0.1 MB | 162 KB | 170 KB |
- | `vdb7-swift` | `swift` | 75 | 0.7 MB | 0.1 MB | 67 KB | 71 KB |
- | `vdb7-docker` | `docker` | 74 | 0.3 MB | 0.1 MB | 23 KB | 24 KB |
- | `vdb7-opam` | `opam` | 61 | 0.3 MB | 0.1 MB | 31 KB | 33 KB |
- | `vdb7-bitnami` | `bitnami` | 41 | 0.2 MB | 0.1 MB | 5.1 KB | 4.9 KB |
- | `vdb7-huggingface` | `huggingface` | 40 | 0.2 MB | – | 16 KB | 17 KB |
- | `vdb7-pub` | `pub` | 28 | 0.2 MB | – | 15 KB | 16 KB |
- | `vdb7-bitbucket` | `bitbucket` | 26 | 0.2 MB | – | 12 KB | 13 KB |
- | `vdb7-cran` | `cran` | 13 | 0.1 MB | – | 7.6 KB | 7.6 KB |
- | `vdb7-cocoapods` | `cocoapods` | 6 | 0.1 MB | – | 4.4 KB | 4.2 KB |
- | `vdb7-conda` | `conda` | 2 | 0.1 MB | – | 3.0 KB | 2.7 KB |
- Use `vdb7-full` for container and VM scanning and for anything needing CPE,
- CVE-ID, alias, reference or full-text search. Use `vdb7-app-only` when you
- never scan OS packages. Use a type shard when your workload is scoped to one
- ecosystem.
- `vdb7-app` and `vdb7-app-only` are not the same artifact and are not
- interchangeable. `vdb7-app` is the group shard above: `completeness=partial`,
- for the shard store, placed with `vdb db refresh app`. `vdb7-app-only` is a
- complete database built from an app-only ingest: `completeness=full`,
- installed as the main database with `vdb db refresh full --app-only`. Pointing
- `refresh full` at `vdb7-app` is rejected by design.
- Every shard in this build passed the match-set gate, meaning `CLEAN` on the
- corpus, the prefix probes and the absent-type checks: a shard returns exactly
- what the full database returns for the types it serves.
- ### Artifact URLs
- Every artifact is published to GHCR as an OCI image carrying the two database
- layers plus a `vdb.meta` config, and mirrored to the `AppThreat/vdb` Hugging
- Face dataset under a `v7-<artifact>/` directory.
- ```text
- ghcr.io/appthreat/vdb7-full:v7.0.x-xz app+OS, everything incl. CPE
- ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz application ecosystems only
- ghcr.io/appthreat/vdb7-app-extended:v7.0.x-xz + metadata tables
- ghcr.io/appthreat/vdb7-app-2y:v7.0.x-xz NVD from 2024
- ghcr.io/appthreat/vdb7-app-2y-extended:v7.0.x-xz NVD from 2024 + metadata
- ghcr.io/appthreat/vdb7-app-10y:v7.0.x-xz NVD from 2016
- ghcr.io/appthreat/vdb7-app-10y-extended:v7.0.x-xz NVD from 2016 + metadata
- ghcr.io/appthreat/vdb7-app:v7.0.x-xz app group SHARD (partial)
- ghcr.io/appthreat/vdb7-npm:v7.0.x-xz one purl type
- ghcr.io/appthreat/vdb7-pypi:v7.0.x-xz
- hf download --repo-type dataset AppThreat/vdb v7-app-only/data.vdb7.zst
- hf download --repo-type dataset AppThreat/vdb v7-npm/data.vdb7 # raw .vdb7
- ```
- `v7-xz` and `v7-zst` track the newest build, `latest-xz` and `latest-zst` are
- aliases for them, and `v7.0.x-xz` and `v7.0.x-zst` pin the release line. Swap
- the suffix for `-zst` to fetch the zstd variant. On Hugging Face, `full`,
- `deb`, `rpm`, `apk`, `app-10y` and `app-10y-extended` are published compressed
- only; smaller artifacts are also mirrored as raw `.vdb7` files.
- ```shell
- oras pull ghcr.io/appthreat/vdb7-app-only:v7-xz # newest build
- oras pull ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz # pinned release line
- ```
- ### Scope variants
- Beyond `vdb7-full` and `vdb7-app-only`, the build publishes five app-scope
- variants that differ only in how much they ingest. All are complete databases
- (`completeness=full`), installable as the main database. `--flavor` names are
- the published repo suffixes, so `--flavor app-extended` installs
- `ghcr.io/appthreat/vdb7-app-extended:v7.0.x-xz` (or your mirror's equivalent:
- the non-env flavors inherit their registry and tag from `VDB_DATABASE_URL`).
- | Artifact | NVD from | Metadata tables | Install with |
- | :---------------------- | :------- | :-------------- | :---------------------------------------------- |
- | `vdb7-app-only` | 2020 | no | `vdb db refresh full --app-only` |
- | `vdb7-app-extended` | 2020 | yes | `vdb db refresh full --flavor app-extended` |
- | `vdb7-app-2y` | 2024 | no | `vdb db refresh full --flavor app-2y` |
- | `vdb7-app-2y-extended` | 2024 | yes | `vdb db refresh full --flavor app-2y-extended` |
- | `vdb7-app-10y` | 2016 | no | `vdb db refresh full --flavor app-10y` |
- | `vdb7-app-10y-extended` | 2016 | yes | `vdb db refresh full --flavor app-10y-extended` |
- `--image <oci-ref>` still overrides the URL for any single refresh, e.g. to
- pin a different release line: `vdb db refresh full --flavor app-10y --image
- ghcr.io/appthreat/vdb7-app-10y:v7-xz`.
- "Metadata tables" means the build ran with `--include-metadata`, so the
- metadata search APIs have data to answer from. Without them those APIs return
- nothing rather than failing, which is why the `-extended` variants exist: they
- are the pre-built answer to "build locally with `--include-metadata`".
- The 10-year variants cover NVD back to 2016 instead of 2020. That widens
- CVE-ID and CPE coverage for older advisories; it does not change how OSV
- ecosystem data is ingested.
- There is no `vdb7-app-2y` on the v7 line yet, and no OS-inclusive `-extended`
- or `-10y` variant — the 6.7.x line published both. Only the app-scope variants
- above are built for v7.
- If you run your own workflow, keep the same naming pattern internally if it
- helps downstream tooling, but publish from infrastructure you control.
- ## CLI usage
- The CLI is flag-based, with three leading-token subcommands: `vdb split` and
- `vdb verify` for the splitter, and `vdb db` for getting and refreshing data.
- ```text
- usage: vdb [-h] [--clean] [--cache] [--cache-os] [--only-osv] [--only-aqua]
- [--only-ghsa] [--include-metadata] [--no-split] [--quiet]
- [--search SEARCH] [--search-text SEARCH_TEXT]
- [--search-alias SEARCH_ALIAS] [--search-reference SEARCH_REFERENCE]
- [--search-package-name SEARCH_PACKAGE_NAME]
- [--search-symbol SEARCH_SYMBOL] [--search-packages SEARCH_PACKAGES]
- [--batch-size BATCH_SIZE] [--list-malware] [--bom BOM_FILE]
- [--print-vdb-metadata] [--custom-data CUSTOM_DATA]
- options:
- -h, --help show this help message and exit
- --clean Clear the vulnerability database cache from platform
- specific user_data_dir.
- --cache Cache vulnerability information in platform specific
- user_data_dir.
- --cache-os Cache OS vulnerability information in platform specific
- user_data_dir.
- --only-osv Use only OSV as the source. Use with --cache.
- --only-aqua Use only Aqua vuln-list as the source. Use with --cache.
- --only-ghsa Use only recent ghsa as the source. Use with --cache.
- --include-metadata Populate extended metadata tables for text, alias,
- reference, symbol, severity, and source searches.
- Increases index database size.
- --no-split Skip splitting the built database into per-type shards
- under the shards directory (VDB_SHARDS_DIR). Splitting
- runs by default after a successful --cache/--cache-os
- build.
- --quiet Suppress logo, logs, and cache progress output.
- --search SEARCH Search for the package or vulnerability ID (CVE, GHSA,
- ALSA, DSA, etc.) in the database. Use purl, cpe, or git
- http url.
- --search-text SEARCH_TEXT
- Perform metadata/full-text search across vulnerability
- descriptions, aliases, references, and affected symbols.
- --search-alias SEARCH_ALIAS
- Search vulnerability aliases such as GHSA, OSV, or
- vendor advisory identifiers.
- --search-reference SEARCH_REFERENCE
- Search reference URLs and reference text in
- vulnerability metadata.
- --search-package-name SEARCH_PACKAGE_NAME
- Search vulnerability metadata by package name or
- namespace/name.
- --search-symbol SEARCH_SYMBOL
- Search affected functions or modules captured in
- vulnerability metadata.
- --search-packages SEARCH_PACKAGES
- Path to a JSON file containing a list of package
- locators to search in bulk. Each item may include purl,
- cpe, url, alias, package_name, or search.
- --batch-size BATCH_SIZE
- Batch size to use with --search-packages.
- --list-malware List latest malwares with CVE ID beginning with MAL-.
- --bom BOM_FILE Search for packages in the CycloneDX BOM file.
- --print-vdb-metadata Display metadata about the current vdb in
- user_data_dir.
- --custom-data CUSTOM_DATA
- Path to directory containing custom vulnerability data
- (JSON/YAML/TOML) to override/augment results.
- ```
- ### Searching
- ```shell
- vdb --search pkg:pypi/xml2dict@0.2.2
- vdb --search pkg:pypi/xml2dict # a purl prefix
- vdb --search "pkg:swift/github.com/vapor/vapor@4.39.0"
- vdb --search "pkg:swift/vapor/vapor@4.89.0" # short form
- vdb --search "cpe:2.3:a:npm:gitblame:*:*:*:*:*:*:*:*"
- vdb --search "npm:gitblame:0.0.1" # colon-separated
- vdb --search CVE-2024-25169
- vdb --search CVE-2025-% # wildcard
- vdb --search "https://github.com/electron/electron" # git url
- vdb --bom bom.json # CycloneDX SBOM
- ```
- Metadata searches need a database built with `--include-metadata`:
- ```shell
- vdb --search-text "deserialization parser"
- vdb --search-alias GHSA-gh-9999
- vdb --search-reference github.com/example/widget-ui
- vdb --search-package-name widget-ui
- vdb --search-symbol parse_payload
- ```
- Bulk search from a JSON file, optionally streamed in batches:
- ```shell
- vdb --search-packages packages.json
- vdb --search-packages packages.json --batch-size 25
- ```
- ```json
- [
- { "purl": "pkg:pypi/requests@2.31.0" },
- { "url": "https://github.com/pallets/flask" },
- { "alias": "GHSA-xxxx-yyyy-zzzz" },
- { "package_name": "flask" }
- ]
- ```
- To list malware entries:
- ```shell
- vdb --list-malware
- ```
- A search against an empty database exits 1 with an explicit message rather than
- printing no results. Wire CI to treat that as scanner down, not as clean.
- ## Custom vulnerability data
- VDB can load custom data from a local directory at runtime, to add private
- vulnerabilities that are not public, or to override false positives by marking
- specific versions as unaffected. Data must follow the CVE 5.2 JSON schema, in
- `.json`, `.yaml`, `.yml` or `.toml`.
- ```shell
- vdb --search pkg:npm/my-lib@1.0.0 --custom-data /path/to/custom/vulns
- ```
- Custom data is operator-controlled policy input. It can override official
- results, so in a hosted or automated integration treat the directory as a
- sensitive input path.
- ### Adding a private vulnerability
- Use the `cna` container, since you are defining a new record.
- ```yaml
- dataType: CVE_RECORD
- dataVersion: "5.2"
- cveMetadata:
- cveId: PRIVATE-2025-001
- assignerOrgId: 00000000-0000-4000-8000-000000000000
- state: PUBLISHED
- datePublished: "2025-01-01T00:00:00Z"
- dateUpdated: "2025-01-01T00:00:00Z"
- containers:
- cna:
- providerMetadata:
- orgId: 00000000-0000-4000-8000-000000000000
- descriptions:
- - lang: en
- value: "Private vulnerability in internal library"
- affected:
- - vendor: internal
- product: my-lib
- packageName: my-lib
- packageURL: pkg:npm/my-lib
- versions:
- - version: "1.0.0"
- status: affected
- versionType: semver
- lessThan: "2.0.0"
- ```
- ### Overriding a false positive
- Use an ADP (Authorized Data Publisher) container to append to or dispute
- existing data. When a CVE ID and purl combination exists in your custom data,
- VDB ignores the official entry and uses yours.
- ```yaml
- dataType: CVE_RECORD
- dataVersion: "5.2"
- cveMetadata:
- cveId: CVE-2023-9999
- assignerOrgId: 00000000-0000-4000-8000-000000000000
- state: PUBLISHED
- containers:
- adp:
- - providerMetadata:
- orgId: 00000000-0000-4000-8000-000000000000
- shortName: "MySecTeam"
- descriptions:
- - lang: en
- value: "Override to mark specific version as unaffected"
- affected:
- - product: requests
- packageName: requests
- packageURL: pkg:pypi/requests
- versions:
- - version: "2.31.0"
- status: unaffected
- versionType: semver
- ```
- ## Using the library
- `vulnerability-db` is designed to be embedded in Python applications. See
- [INTEGRATION.md](./INTEGRATION.md) for schemas, direct-SQL patterns and the
- full result shape.
- ```python
- import json
- import os
- from vdb.lib import config, db as db_lib, search
- from vdb.lib.orasclient import download_image
- # Prefer an organization-controlled URL. The upstream default is for local
- # testing and bootstrap.
- DB_URL = os.getenv("VDB_APP_ONLY_DATABASE_URL", config.VDB_APP_ONLY_DATABASE_URL)
- if db_lib.needs_update(days=1):
- download_image(DB_URL, config.DATA_DIR)
- # Optional: override results with local YAML/JSON definitions
- # search.load_custom_data("/path/to/custom_vulns")
- results = search.search_by_any("pkg:pypi/xml2dict@0.2.2", with_data=True)
- output = []
- for res in results:
- vuln = {"cve_id": res["cve_id"], "fixed_in": res["fix_version"]}
- if res.get("source_data"):
- # source_data is a plain dict, ready for json.dumps
- vuln["cve_data"] = res["source_data"]
- output.append(vuln)
- print(json.dumps(output, indent=2))
- ```
- `source_data` is a dict, not a Pydantic model, and has no `.model_dump()`.
- Snippets from the 6.x era that call
- `res["source_data"].model_dump(mode="json")` raise `AttributeError` on v7.
- For production, point `VDB_APP_ONLY_DATABASE_URL` or `VDB_DATABASE_URL` at your
- own workflow's artifacts so application instances do not depend on a
- third-party refresh job for security-relevant data.
- ### Bulk and filtered searches
- ```python
- from vdb.lib import search
- filters = {
- "severity_threshold": "HIGH",
- "sources": ["osv", "github"],
- "exclude_malware": True,
- "package_ecosystem": "pypi",
- "page_size": 25,
- }
- package_results = search.search_packages(
- [
- {"purl": "pkg:pypi/requests@2.31.0"},
- {"url": "https://github.com/pallets/flask"},
- {"cpe": "cpe:2.3:a:npm:lodash:4.17.20:*:*:*:*:*:*:*"},
- ],
- with_data=False,
- filters=filters,
- )
- for pkg in package_results:
- print(pkg["locator"], pkg["result_count"], pkg["max_severity"])
- ```
- For very large package sets, stream them:
- ```python
- for batch in search.search_packages_batched(packages, batch_size=100, with_data=False):
- for pkg in batch:
- print(pkg["locator"], pkg["result_count"])
- ```
- CycloneDX BOMs can be summarised or expanded:
- ```python
- summary = search.search_bom_summary("bom.json", filters={"severity_threshold": "MEDIUM"})
- detailed = search.search_bom_detailed("bom.json", with_data=True)
- ```
- For large SBOMs, `search_by_cdx_bom` yields batches so peak memory stays
- bounded:
- ```python
- for batch in search.search_by_cdx_bom("bom.json", with_data=True):
- for res in batch:
- ...
- ```
- ### Text, alias, reference and symbol search
- ```python
- search.search_by_alias("GHSA-xxxx-yyyy-zzzz", with_data=False)
- search.search_by_reference("github.com/pallets/flask", with_data=False)
- search.search_by_package_name("flask", with_data=False)
- search.search_by_symbol("parse_payload", with_data=False)
- search.search_full_text("deserialization parser", with_data=False)
- ```
- These use the metadata tables in `data.index.vdb7`. Locator searches work
- without metadata, so choose a `--include-metadata` database when your
- integration depends on these APIs or on metadata filters. On a database without
- metadata they fail safe by returning no metadata matches, which means an empty
- result there tells you nothing about the package.
- ### Partial (shard) databases
- A database whose `vdb.meta` declares `"completeness": "partial"` is a per-type
- shard. Purl lookups work normally and return exactly the full database's
- matches for the purl types the shard serves. Entry points whose queries span
- purl types (`search_by_cpe_like`, `search_by_cve`, `search_by_alias`,
- `search_by_reference`, `search_full_text`, `search_by_symbol`,
- `latest_malware`) raise `vdb.lib.search.PartialDatabaseError` rather than
- silently under-report.
- For a purl type no local shard serves, `search_by_purl_like` raises
- `vdb.lib.search.ShardCoverageError` instead of returning an empty list that
- would read as "no vulnerabilities". Batch and BOM searches instead mark the
- component `coverage_gap: true` and report a `coverage` block, so one missing
- shard does not fail a whole scan.
- ### Deployment notes
- Set `VDB_HOME` before importing `vdb.lib.config`. Path and build-scope
- variables are read at import time, so setting them later in a long-running
- process has no effect.
- For read-only deployments where the `.vdb7` files are not modified while the
- process runs, set `VDB_SQLITE_IMMUTABLE=true`.
- ## Environment variables
- Boolean toggles treat `true` or `1` as enabled. Set path and build-scope
- variables before importing `vdb.lib.config`.
- ### Paths and local storage
- | Variable | Default | Used by | Description |
- | :----------------- | :------------------------------- | :----------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `VDB_HOME` | Platform user data directory | CLI and library | Directory containing `data.vdb7`, `data.index.vdb7` and `vdb.meta`. |
- | `VDB_CACHE` | Platform user cache directory | `vdb --cache-os`, `AquaSource` | Cache directory. If `$VDB_CACHE/vuln-list.zip` exists it is used instead of downloading. |
- | `VDB_TEMP_DIR` | System temp directory | SQLite setup | Directory for SQLite temporary files during large builds and `VACUUM`. Use a partition with room for app+OS builds. |
- | `VDB_SHARDS_DIR` | `$VDB_HOME/shards` | Shard-mode search | Local type-shard store: one `<shard>/` directory per shard, each with `data.vdb7`, `data.index.vdb7` and a v2 `vdb.meta`. The connected main database participates as a shard when its own meta declares `completeness: "partial"`. |
- | `VDB_AUTO_FETCH` | unset | Shard-mode search | Opt-in on-demand shard fetch. When enabled, a shard-mode purl search needing a missing shard fetches it from the registry recorded in a local shard's `vdb.meta` siblings. Off by default so scans never reach for the network implicitly. |
- | `VDB_SHARD_FANOUT` | `sequential` | Shard-mode search | Multi-shard fan-out strategy. `sequential` runs the canonical per-shard search path; `attach` UNION-ALLs attached shard indexes for the index sweep. Both are gated by the multi-shard match-set gate. |
- | `VDB_ZSTD_BIN` | unset (search `PATH` for `zstd`) | `-zst` artifact refresh | Explicit zstd binary for unpacking `-zst` artifacts on Python < 3.14 (3.14+ uses the stdlib `compression.zstd` module). An absolute path for bundled or air-gapped deployments, or a bare name resolved against `PATH`. A value that resolves to nothing leaves `zst` refreshes refused with both remedies named; `xz` always works. |
- ### Pre-built database downloads
- | Variable | Default | Used by | Description |
- | :-------------------------- | :------------------------------------------ | :------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `VDB_APP_ONLY_DATABASE_URL` | `ghcr.io/appthreat/vdb7-app-only:v7.0.x-xz` | `vdb db refresh full --app-only`, MCP auto-download | OCI image URL for the default app-only database. `vdb7-app-only` (a complete app-scope database), not `vdb7-app` (the partial group shard). Override to consume an internally published artifact. |
- | `VDB_DATABASE_URL` | `ghcr.io/appthreat/vdb7-full:v7.0.x-xz` | `vdb db refresh full`, `vdb db refresh full --image <oci-ref>` | OCI image URL for the app+OS database. Must be a v7 artifact; a v6 image is rejected. Prefer the `-xz` variant: it unpacks unconditionally, while `-zst` needs a zstd decompressor (Python 3.14+ or a `zstd` binary on PATH). |
- ### Source and feed selection for builds
- | Variable | Default | Used by | Description |
- | :----------------------------- | :------------------------------- | :------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `NVD_START_YEAR` | `2022` | NVD/vuln-list conversion, distro feeds | Start year for NVD-style CVE data and the Linux distro feeds. Older years increase coverage, build time and database size. Does not apply to application ecosystem advisories, because a distro fix moves the whole release forward while a lockfile does not. |
- | `VDB_APP_ECOSYSTEM_START_YEAR` | unset (no floor) | OSV application ecosystem feeds | Re-imposes a start year on the application ecosystems (npm, PyPI, Maven, Go, crates.io, NuGet, RubyGems, Hex, Packagist, Pub, CRAN, SwiftURL, Julia, GitHub Actions). A year, or `nvd` to follow `NVD_START_YEAR`. Set it only to trade coverage for a smaller artifact. |
- | `GITHUB_TOKEN` | unset | GitHub advisory ingestion | Token for the GitHub GraphQL API. Avoid printing this value in logs. |
- | `GITHUB_GRAPHQL_URL` | `https://api.github.com/graphql` | GitHub advisory ingestion | Alternate GitHub GraphQL endpoint, for testing or enterprise proxies. |
- | `GITHUB_PAGE_COUNT` | `2` | GitHub advisory ingestion | Number of GitHub advisory GraphQL pages to fetch during a full refresh. |
- | `NPM_PAGE_COUNT` | `2` | npm advisory configuration | Number of npm advisory pages to fetch where npm ingestion is used. |
- | `OSV_INCLUDE_FUZZ` | unset | OSV ingestion | Include the Linux, OSS-Fuzz and Android OSV feeds that are excluded by default to reduce false positives. |
- | `OSV_EXCLUDE_MALWARE` | unset | OSV conversion | Exclude OSV malware advisories whose identifiers start with `MAL`. |
- | `VDB_OSV_STORE_BATCH_SIZE` | `100` | OSV ingestion | Converted OSV records stored per database batch. Invalid values fall back to `100`; minimum 1. |
- | `VDB_MAX_AFFECTED_PER_BLOB` | `32` | CVE-5 blob storage | Maximum `affected` entries merged into one source blob per CVE; further entries spill into more blobs so blob size and per-row hydration cost stay flat as CVE fan-out grows. `0` disables merging; invalid values fall back to `32`. |
- | `VDB_INCLUDE_METADATA` | unset | CLI, storage, search metadata indexes | Populate extended metadata tables. Equivalent to `--include-metadata`. |
- | `VDB_METADATA_*` | unset | `vdb.meta` creation | Adds custom build metadata to `vdb.meta`; the prefix is stripped and the key lowercased. `true`/`1` and `false`/`0` are stored as booleans. |
- ### OS distro filtering for builds
- | Variable | Default | Used by | Description |
- | :------------------------- | :------ | :-------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `VDB_IGNORE_OS` | unset | OSV source configuration | Skip the OSV operating-system feeds added by default. Use app-only workflows for the smallest app database. |
- | `VDB_IGNORE_<DISTRO>` | unset | vuln-list and selected OSV distro feeds | Exclude distro-specific data. Static OSV toggles are `VDB_IGNORE_ALMALINUX`, `VDB_IGNORE_ALPINE`, `VDB_IGNORE_REDHAT`, `VDB_IGNORE_DEBIAN`, `VDB_IGNORE_ROCKYLINUX`, `VDB_IGNORE_MAGEIA`, `VDB_IGNORE_ALPAQUITA`, `VDB_IGNORE_MINIMOS`, `VDB_IGNORE_UBUNTU`, `VDB_IGNORE_AZURE_LINUX` and `VDB_IGNORE_SEAL`; vuln-list also supports the distro keys in `LINUX_DISTRO_VULN_LIST_PATHS` (including `VDB_IGNORE_ORACLE` and `VDB_IGNORE_BOTTLEROCKET`). SUSE and openSUSE are opt-in instead. Debian, Ubuntu, Red Hat, SUSE, openSUSE and Azure Linux data comes from the OSV feeds only. |
- | `VDB_EXCLUDE_<DISTRO>` | unset | vuln-list filtering | Alias for excluding vuln-list distro paths. |
- | `VDB_INCLUDE_<DISTRO>` | unset | vuln-list filtering | Force-include distro paths from `LINUX_DISTRO_VULN_LIST_PATHS`, such as `VDB_INCLUDE_ALPINE=true`. `VDB_INCLUDE_SUSE=true` and `VDB_INCLUDE_OPENSUSE=true` are separate: they enable the OSV SUSE and openSUSE feeds, the only source of SUSE family data, which are off by default. They have the heaviest errata fan-out of any feed, so they stay opt-in for local builds; the published OS databases enable both. |
- | `VDB_IGNORE_LINUX_KERNEL` | `true` | Storage | Skip `pkg:generic/linux` records. The kernel feed dominates the database (around 130k rows) with commit-sha version ranges that semantic-version lookups cannot match. |
- | `VDB_INCLUDE_LINUX_KERNEL` | unset | Storage | Opt back in to storing Linux kernel CVEs. Overrides `VDB_IGNORE_LINUX_KERNEL`. |
- ### Output and SQLite tuning
- | Variable | Default | Used by | Description |
- | :------------------------ | :------- | :-------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `VDB_QUIET` | unset | CLI | Suppress logo, logs and cache progress output. Equivalent to `--quiet`. |
- | `VDB_PROGRESS_INTERVAL` | `10000` | Source progress callbacks | Minimum records between progress messages. Invalid values fall back to `10000`; minimum 1. |
- | `VDB_SQLITE_IMMUTABLE` | unset | Search database connections | Open file-backed search databases with SQLite's immutable URI option, for read-only deployments where `.vdb7` files are not modified while the process runs. |
- | `VDB_SQLITE_CACHE_SIZE` | `-65536` | SQLite setup | Value passed to `PRAGMA cache_size`. The default is about 64 MiB using SQLite's negative-KiB convention. |
- | `VDB_SQLITE_JOURNAL_MODE` | `DELETE` | SQLite setup | Value passed to `PRAGMA journal_mode`. Valid: `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, `WAL`, `OFF`. |
- | `VDB_SQLITE_SYNCHRONOUS` | `OFF` | SQLite setup | Value passed to `PRAGMA synchronous`. Valid: `OFF`, `NORMAL`, `FULL`, `EXTRA`, or `0` to `3`. |
- | `VDB_DATA_PAGE_SIZE` | `32768` | Builds and splits | Page size for the data database. Powers of two from `512` to `65536`; invalid values warn and fall back. The 32K default ships around 27% smaller files and halves cold hydration; set `4096` to restore the previous size. |
- | `VDB_INDEX_PAGE_SIZE` | unset | Builds and splits | Page size for the index database. The random-access index file shows no win past the `4096` default, so leaving this unset is the recommendation. |
- | `PYTHONIOENCODING` | unset | Windows startup | If unset on Windows, VDB reconfigures standard streams to UTF-8. |
- ### MCP server
- | Variable | Default | Used by | Description |
- | :------------- | :------ | :--------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
- | `VDB_AGE_DAYS` | `2` | MCP server | Days before the MCP server considers the local database stale and attempts an app-only ORAS download. Non-numeric values are passed through to the freshness check, so prefer an integer string. |
- ## Troubleshooting
- **Database locked, or `apsw.BusyError`.** Check that you are not running two
- `vdb --cache` processes at once. In a multi-threaded application, treat
- connections as read-only wherever possible. Reading one database while a build
- writes another can also raise this.
- **Disk space during a build or download.** The extracted full database is
- around 15.6 GB, and SQLite needs significant temporary space for index creation
- and `VACUUM` during a build. Point `VDB_TEMP_DIR` at a partition with room:
- ```shell
- export VDB_TEMP_DIR=/mnt/large_volume/vdb_temp
- vdb --cache-os
- ```
- **ORAS download failures.** Install the extra with
- `pip install appthreat-vulnerability-db[oras]`. Firewalls may block `ghcr.io`,
- in which case download manually. If your environment restricts outbound access
- or you need stronger provenance, publish databases from your own network and
- point clients at those artifacts.
- **A download that succeeds but leaves no database.** Check the compression
- suffix. `xz` unpacks unconditionally; a `-zst` URL unpacks only when a zstd
- decompressor is available (Python 3.14+ or a `zstd` binary on PATH) — the
- refresh refuses a zst fetch without one before downloading anything, naming
- both remedies.
- **`UnicodeEncodeError` on Windows.** VDB forces UTF-8 for stdout and stderr, but
- the terminal also has to be configured for it (`chcp 65001`).
- ## Model Context Protocol (MCP) server
- See the [MCP server readme](./packages/mcp-server-vdb/README.md).
- The server exposes structured tool results, resource templates such as
- `cve://{id}` and `purl://{purl}`, concrete resources such as `vdb://metadata`
- and `vdb://health`, bulk SBOM tools, prompts, and filter-aware search options
- covering severity thresholds, source filters, malware flags, package scope and
- pagination.
- ## Reading `.vdb7` files in other languages
- `.vdb7` files are standard SQLite databases; any modern SQLite library can read
- them. The tables, indexes and every query vdb itself runs are documented in
- [docs/DATABASE.md](./docs/DATABASE.md), and the design of the shard store,
- search fan-out and on-demand downloads in [docs/DESIGN.md](./docs/DESIGN.md).
- For agents and scripts doing aggregation or bulk export, querying directly is
- reasonable:
- ```bash
- sqlite3 "file:$VDB_HOME/data.index.vdb7?mode=ro" \
- "SELECT * FROM cve_index WHERE purl_prefix LIKE 'pkg:npm/react%';"
- ```
- Note that direct SQL sees none of the coverage, staleness or empty-database
- guards the library applies. On a shard, a purl of an uncovered type returns no
- rows, which is indistinguishable in SQL from clean. Prefer the library APIs
- when the answer will be reported to someone.
- ## For AI agents
- [SKILL.md](./SKILL.md) is the guide for using `vdb`, and
- [AGENTS.md](./AGENTS.md) is the guide for changing it. Both are written for
- agents. The essentials:
- Download a database before searching. `vdb db refresh full --app-only` for
- application packages, `vdb db refresh full` for OS packages. The refresh is
- staged and validated before it replaces the local database. Prefer a URL from
- the user's own workflow or mirror over the AppThreat-hosted default.
- Use the Python API for structured output. The CLI prints human-readable tables
- and has no `--json` flag on the search commands.
- Treat an empty result as a question, not an answer. Check that the search
- exited 0, that every requested purl type was covered, and whether any answering
- shard was stale. All three are reported.
- ## Local development
- Set up [uv](https://docs.astral.sh/uv/), then:
- ```shell
- uv sync --all-extras --dev
- uv run vdb --help
- uv run pytest
- uv run ruff check vdb test contrib
- ```
- To compare search latency while changing schemas, indexes or pragmas:
- ```shell
- uv run python contrib/bench_search.py \
- --data vdb_data/data.vdb7 \
- --index vdb_data/data.index.vdb7 \
- --query 'pkg:pypi/requests@2.31.0' \
- --repeat 5
- ```
- Pass `--with-data` to include CVE source-data decoding, or `--json` for
- machine-readable output.
- ## Citation
- ```text
- @misc{vdb,
- author = {Team AppThreat},
- month = Feb,
- title = {{AppThreat vulnerability-db}},
- howpublished = {{https://huggingface.co/datasets/AppThreat/vdb}},
- year = {2025}
- }
- ```
- ## License
- MIT
Ratings and comments
Sign in to rate this skill.
Sign inFree to use. If it’s helped, pass it on.