Guidance for AI coding agents and automated contributors working on
appthreat-vulnerability-db.
If you are using VDB rather than changing it, read SKILL.md instead. This file is about working on the code.
vdb builds and queries an offline SQLite vulnerability database from
AppThreat vuln-list, OSV, NVD, GitHub and npm. It is used as a CLI and as a
Python library by downstream scanners.
Four goals, in the order they win arguments:
Accurate package locator and version matching. A wrong answer here becomes a wrong security decision downstream, so accuracy beats size and speed.
Fast offline lookup from pre-built .vdb7 files, with no network access
during a scan.
Reproducible builds. App-only and app+OS databases, plus per-type shards, from a fresh build every time. There is no migration path and none is wanted.
Bounded resource use on large feeds. Memory and disk are first-class constraints, not afterthoughts.
vdb/cli.py CLI entry point, cache/download/search orchestration
vdb/lib/config.py environment-driven source, path and database config
vdb/lib/db.py SQLite connections, schema, pragmas, finalization
vdb/lib/shard_store.py shard store discovery, fan-out, coverage, staleness
vdb/lib/shard_split.py the splitter: full database to per-type shards
vdb/lib/db_cmd.py the `vdb db` subcommand
vdb/lib/cve.py CVE 5.2 conversion and storage
vdb/lib/osv.py OSV feed ingestion and conversion
vdb/lib/aqua.py AppThreat/Aqua vuln-list ingestion, incl. OS distros
vdb/lib/gha.py GitHub advisory ingestion
vdb/lib/nvd.py NVD ingestion and the shared CVE-5 conversion
vdb/lib/cpe.py CPE parsing, normalisation, separator variants
vdb/lib/vers.py the single version comparison implementation
vdb/lib/search.py query APIs and result materialization
vdb/lib/search_index.py metadata indexes for text/alias/reference/symbol
test/ pytest suite and small fixture data
contrib/ measurement harnesses and their findings documents
docs/ reference docs: DATABASE.md (schema + queries), DESIGN.md (shard client architecture), BENCHMARKS.md
packages/mcp-server-vdb/ MCP server, depends on the library API
The workflow repository that builds public artifacts is checked out separately,
usually at ../vdb.
Gather context before changing behaviour. Trace the call path from the CLI
or the source converter through CVESource.store(), vdb.lib.db and search before
editing. Ingestion has more layers than it looks: the year floor, for example,
is enforced in three separate places, and changing one of them just moves where
a record is dropped.
Measure before you claim. Never state a number you did not run. If a change is justified by a count, run the count and put it in the commit message. If you skipped a measurement, say so rather than estimating.
Prefer small fixes with regression tests. Most bugs here involve data volume or a specific feed shape. Add a deterministic fixture or a synthetic harness rather than a test that needs a live download.
Do not download full feeds unless necessary. Prefer
$VDB_CACHE/vuln-list.zip, local zips, and the existing test/data/*.json
files. When a feed-side measurement genuinely needs the real data, download the
one ecosystem zip you need, not all of them.
Bind a scratch database before constructing a source. The safe order is
db.get(":memory:", ":memory:") first, then the source. A source constructed
first binds whatever database is already bound, which in a scratch script can
be the real one.
Treat an existing production database as read-only. Open it with
mode=ro&immutable=1. Never call clear_all() on it, never rebuild in place,
and do not read one database while a build of your own is writing another.
Assume artifacts are rebuilt from scratch. Do not add schema migration code or legacy-write compatibility unless asked. Search compatibility with older downloaded databases is not a requirement for storage-layout changes.
Treat database size and transient disk use as requirements. When changing SQLite logic, test row counts, distinct hashes, file sizes and temp/journal behaviour. A change that grows the published artifact is a decision for a human, not a side effect.
Do not swallow exceptions around storage. A bare except: pass in an
ingestion loop turns a data-corruption bug into missing rows that nothing
reports. If you must catch broadly, make the failure visible and make sure the
batch cannot grow unboundedly across failures.
Avoid broad formatting churn. Preserve existing style. Reformat only lines you changed.
Never expose tokens. Do not print GITHUB_TOKEN, HF_TOKEN, registry
credentials or environment dumps in tests or logs.
Keep environment variables documented. The README reference must stay in
sync with vdb/lib/config.py, vdb/lib/db.py, the ingestion modules and the
MCP server. Current groups: paths (VDB_HOME, VDB_CACHE, VDB_TEMP_DIR,
VDB_SHARDS_DIR), shard behaviour (VDB_AUTO_FETCH, VDB_SHARD_FANOUT),
download URLs (VDB_DATABASE_URL, VDB_APP_ONLY_DATABASE_URL), source
controls (NVD_START_YEAR, VDB_APP_ECOSYSTEM_START_YEAR, GITHUB_TOKEN,
GITHUB_GRAPHQL_URL, GITHUB_PAGE_COUNT, NPM_PAGE_COUNT,
OSV_EXCLUDE_MALWARE, OSV_INCLUDE_FUZZ, VDB_OSV_STORE_BATCH_SIZE,
VDB_MAX_AFFECTED_PER_BLOB), metadata and progress (VDB_INCLUDE_METADATA,
VDB_METADATA_*, VDB_QUIET, VDB_PROGRESS_INTERVAL), SQLite
(VDB_SQLITE_IMMUTABLE, VDB_SQLITE_CACHE_SIZE, VDB_SQLITE_JOURNAL_MODE,
VDB_SQLITE_SYNCHRONOUS, VDB_DATA_PAGE_SIZE, VDB_INDEX_PAGE_SIZE), distro
filters (VDB_IGNORE_OS, VDB_IGNORE_*, VDB_EXCLUDE_*, VDB_INCLUDE_*,
VDB_IGNORE_LINUX_KERNEL, VDB_INCLUDE_LINUX_KERNEL), and MCP
(VDB_AGE_DAYS).
Worth knowing before you “fix” one of these.
An empty vers string matches every version. Records with no version range do
apply to all versions.
A non-numeric version sorts above numeric ones. This is what keeps an npm
@latest dist-tag from matching every advisory with an upper bound.
Chainguard and Wolfi advisories with no published fix are stored as affecting every version rather than dropped.
cve_index uses ON CONFLICT IGNORE, so the first writer of a
(cve_id, vers, purl_prefix) wins. Reordering sources changes which row
survives, database-wide. It is not a performance-only change.
Per-type ecosystem shards overlap the group shards on purpose. The npm shard
and the app shard both serve npm. They are separate views, not a partition.
Application ecosystem advisories are not year-filtered. That is deliberate, and
NVD_START_YEAR applies only to NVD-style data and the distro feeds.
uv sync --all-extras --dev
uv run pytest
uv run ruff check vdb test contrib
uv run vdb --help
Focused runs while working on ingestion or search:
uv run pytest test/test_source.py -k osv
uv run pytest test/test_source.py -k aqua
uv run pytest test/test_search_advanced.py
uv run pytest test/test_reachability.py
uv run pytest test/test_db.py test/test_db_size.py
When adding a database regression test, prefer :memory: databases via
db.get(":memory:", ":memory:") unless file size itself is under test. Always
call db.reset_connections() before switching database paths.
AquaSource uses a cached zip at $VDB_CACHE/vuln-list.zip when present.
export VDB_HOME=/tmp/vdb-home
export VDB_CACHE=/tmp/vdb-cache
export VDB_TEMP_DIR=/tmp/vdb-home/tmp
mkdir -p "$VDB_HOME" "$VDB_CACHE" "$VDB_TEMP_DIR"
For tests, build a small zip with only representative paths, then monkeypatch
config.CACHE_DIR at it:
vuln-list-main/debian/CVE-2026-0001.json
vuln-list-main/nvd/2026/CVE-2026-0002.json
The build-vdb7.yml workflow in the sibling AppThreat/vdb repository, using
the build-and-upload-vdb7 action for the full and app images and
split-and-upload-vdb7 to split, gate and publish the per-type shards. The
build action sets VDB_HOME, VDB_CACHE, NVD_START_YEAR and
GITHUB_PAGE_COUNT, creates vdb_cache/vuln-list.zip, then runs:
python vulnerability-db/vdb/cli.py --cache-os
Full-database jobs add --no-split, because the split-and-upload action
re-splits. The workflow’s include-metadata input adds --include-metadata.
App-only variants set VDB_IGNORE_OS=true plus selected VDB_IGNORE_* flags.
Builds run on a fresh runner every time, so there is no build isolation to preserve and no incremental state to protect. Optimise for scan-time performance and artifact size, not for rebuild ergonomics.
VDB_EXTENDED_DATABASE_URL, VDB_APP_ONLY_EXTENDED_DATABASE_URL and
USE_VDB_10Y do not exist in v7. Do not document them as live options.
Trace in this order, and add the regression test at the lowest layer that demonstrates the bug:
osv.py, aqua.py, gha.py, nvd.py), including
whether the record survived the feed’s year floor at all.VulnerabilityDetail fields: mii, mie, mai, mae, fixed_location,
package_type.to_purl_vers() in utils.py.CVESource.store5(), for the purl prefix and index row it writes.vdb.lib.vers.vers_compare().Between 4 and 5 sits the most common class of defect in this codebase: the
store writes a locator the lookup never proposes. Both sides construct a
purl_prefix independently, and when the constructions disagree the rows exist
and are unreachable, which reads as “not vulnerable”. contrib/reachability_audit.py
exists to catch exactly this, and a clean match-set gate does not imply
reachability.
Are new source or search paths covered by tests?
Does the change work for a fresh build and a clean download, rather than only for a database that already exists locally?
Are large feeds streamed or processed in bounded batches?
Are SQLite temp files, journals, indexes and VACUUM behaviour considered?
Does the change alter the published artifact’s size or contents? If so, is the number measured and stated?
Can a stored row still be reached by the lookup that is supposed to find it?
Are environment variable names consistent with config.py, and documented?
Are errors handled without silently hiding data corruption?
Are dependency or release workflow changes security-sensitive? See THREAT_MODEL.md.