A working threat model for appthreat-vulnerability-db (vdb), written for
maintainers, security researchers, downstream scanner authors, and agents
reviewing changes across ingestion, storage, search, distribution, MCP
integration and the publishing workflows.
The goal is practical review guidance rather than completeness. vdb takes
large vulnerability feeds, turns them into offline SQLite databases, publishes
those databases through a registry and a dataset, and scanners then use them to
make security decisions. That chain makes integrity, resource use and release
provenance all security-relevant.
One framing point worth stating up front, because it shapes several decisions below. For a vulnerability database, the worst failure is not a crash or a false positive. It is a confident empty answer: a scan that reports nothing because the data was missing, unreachable or not downloaded, and that looks identical to a scan that reports nothing because the software is clean. Several mitigations in this document exist only to keep those two cases distinguishable.
Upstream feeds
AppThreat/Aqua vuln-list zip
OSV ecosystem and distro zips
NVD JSON feeds
GitHub advisories
npm advisories
|
v
Source converters vdb/lib/{aqua,osv,nvd,gha,npm}.py
|
v
CVE 5.2 conversion vdb/lib/cve.py
and storage vdb/lib/db.py
|
+--> data.vdb7 CVE source blobs, keyed by hash, plus
| package locator rows referencing them
+--> data.index.vdb7 purl/version index, optional metadata
|
v
Optional split vdb/lib/shard_split.py
|
+--> shards/<type>/ per-type standalone databases
|
v
Search APIs and CLI
purl / cpe / url / CVE / alias / text / symbol
bulk package and BOM search
shard fan-out with coverage reporting
MCP server
The public build pipeline is a separate, related trust boundary:
GitHub Actions in AppThreat/vdb
checkout vulnerability-db and vuln-list
set VDB_HOME, VDB_CACHE, NVD_START_YEAR, GITHUB_PAGE_COUNT
build app-only or app+OS .vdb7 databases
split into per-type shards and gate them
compress data.vdb7 and data.index.vdb7
push ORAS artifacts to GHCR
upload files to Hugging Face dataset paths
Artifact integrity. Published .vdb7 files, compressed artifacts and
metadata should come from trusted workflows and represent the sources and
settings they claim.
Analysis integrity. Searches should map package locators and versions to vulnerability records accurately, and a row that exists should be reachable by the lookup meant to find it.
Honest absence. Missing data, uncovered purl types and stale shards should be reported as such, never as a clean result.
Resource safety. Large feeds, crafted zips, SQLite temp files and index builds should not cause uncontrolled memory or disk use in expected workflows.
Data handling safety. Custom data, downloaded artifacts, SQLite paths and feed contents should not enable path traversal, SQL injection, unsafe extraction or credential leakage.
Operational clarity. Users should understand which artifact they have, what scope it covers, how old it is, and how to build a narrower internal one.
Published Python packages for appthreat-vulnerability-db. Published .vdb7,
.tar.xz and .zst artifacts. GHCR image tags and Hugging Face dataset paths.
Build metadata in vdb.meta, including source settings, shard type lists and
sibling records. Workflow credentials: GITHUB_TOKEN, HF_TOKEN, registry
credentials and trusted publishing authority. Local paths under VDB_HOME,
VDB_CACHE, VDB_SHARDS_DIR and VDB_TEMP_DIR. Search output consumed by
scanners, CI gates, dashboards, MCP clients and agents. Operator-supplied
custom vulnerability data.
Feed boundary. Upstream feeds are external data. They can be malformed, unexpectedly large, duplicated, internally inconsistent, or crafted if a mirror or source is compromised. Converters must treat feed content as untrusted until converted and validated. Feeds also disagree with each other and with vendors, so a converter that is faithful to its feed can still be wrong about the world; that is a data-quality question, not a bug in the converter.
Cache and zip boundary. AquaSource processes $VDB_CACHE/vuln-list.zip
when present. The zip may come from a workflow, a mirror or a local test. Member
names, sizes and counts are both a parsing and a resource boundary.
SQLite boundary. .vdb7 files are SQLite databases, created during builds
and opened for search. URI handling, temp and journal files, VACUUM, index
creation and file size are all inside the boundary. Databases are built from
scratch with normalized source blobs in cve_source_data and locator rows in
cve_data. Migration of older layouts is deliberately out of scope.
Shard boundary. A shard is a database that answers for some purl types and
not others. Its vdb.meta is the authority on which. A store can legitimately
hold shards from different builds, so freshness is per shard rather than global.
The security-relevant property is that an uncovered type must not answer
“nothing found”.
Artifact publishing boundary. The sibling AppThreat/vdb workflows cross
from source code and feed data into published artifacts: checkout actions,
dependency installation, splitting and gating, compression, ORAS pushes, Hugging
Face uploads and tag management.
Consumer boundary. Downstream tools trust VDB results to make security decisions. A false negative, a poisoned purl mapping, a missing alias, an incorrect version range or a stale artifact all propagate into remediation and policy enforcement.
Custom data boundary. Custom JSON, YAML and TOML vulnerability data is operator-controlled policy input that can override official results. Trusted in local use; a sensitive input path in hosted or automated use.
Malicious feed or mirror operator. Poisons records, alters version and package mappings, or causes denial of service. Note that narrowing a range is as effective as removing a record, and much harder to notice.
Compromised workflow dependency or contributor. Alters published packages or database artifacts.
Attacker controlling custom data or local paths in an integration. Overrides findings, reads files, or triggers unsafe parsing.
Attacker publishing a malicious package. Not an attacker against vdb itself, but the reason the malware feeds exist. Relevant here because names are adversarially chosen: mixed-script homoglyphs, scope confusion and case variants are all deliberate attempts to be stored under one spelling and looked up under another.
Downstream consumer of stale or wrong artifacts. Not malicious, but makes incorrect decisions from misunderstood scope or age.
Accidental operator. Builds a much larger database than intended, publishes from the wrong source, leaks credentials in logs, or uses public artifacts where internal provenance is required.
vdb/lib/{aqua,osv,nvd,gha,npm}.py parse external JSON and zip files. Risks:
malformed JSON causing crashes or silent data loss; extreme record counts
causing memory pressure; zips with many entries or unexpected paths; duplicate
records inflating database size; incorrect source filtering through environment
variables; exception swallowing that hides corruption.
Exception swallowing deserves emphasis because it has bitten this codebase concretely. An ingestion loop that catches broadly and keeps a growing batch turns one poisoned record into an unbounded quadratic loop that stores nothing and never terminates. A single malformed CVSS vector once dropped whole advisories silently. Both were failures of visibility, not of parsing.
Review questions: Is the feed processed in bounded batches? Is the batch cleared even when storage raises? Are include and ignore filters deterministic? Are errors visible enough to diagnose corruption without leaking secrets? Do tests use small fixtures rather than live downloads?
CVESource.store() and store5() convert Vulnerability objects into CVE 5.2
records and SQLite rows. Risks: duplicate rows for equivalent records; repeated
large blobs causing sudden growth; unstable source hashes; incorrect purl
prefixes or version range strings; metadata rows diverging from source data.
Repeated blob growth is mitigated by storing each distinct source record once
in cve_source_data. Source hash stability matters directly: an unstable hash
silently multiplies the artifact.
Two integrity properties are easy to break here. cve_index uses
ON CONFLICT IGNORE, so the first writer of a (cve_id, vers, purl_prefix)
wins and source ordering decides which feed’s row survives. And a stored
locator has to be one the lookup will actually propose. Both sides construct a
purl prefix independently, and when they disagree the rows exist and cannot be
found, which presents to the user as “not vulnerable”.
Review questions: Do row counts match expected affected-package counts? Are
distinct source hashes much lower than locator rows where expected? Can each
stored locator be reached by a scan purl a real SBOM would carry? Does
clear_all() remove all build tables before a rebuild?
A version range is where a security claim becomes a boolean, so its edge cases are security-relevant rather than cosmetic. Three that have caused real wrong answers:
An empty range matches every version, because a record with no version range genuinely applies to all versions. Treating empty as matching nothing under-reports.
A non-numeric version must sort above numeric ones. Coercing an npm @latest
dist-tag to zero puts it below every fixed version, so it matches every
advisory that has an upper bound.
An enumerated version list is usually a sample, not a closed set. A range narrowed to the specific versions someone happened to observe reports nothing for every version in between.
Review questions: Does the change alter which versions match, and is that change measured against a fixture rather than asserted? Is a widened range justified by a statement in the feed, or inferred from the shape of the data?
Search matches purls, CPEs, aliases, references, package names, symbols and text. Risks: SQL injection from dynamic query construction; incorrect version comparison; stale or missing metadata rows; custom data overriding official data unexpectedly; batch queries returning mismatched source data.
Name normalisation is part of this surface, not a usability detail. Locators are lowercased and some feeds publish names percent-encoded, so the lookup has to propose several spellings of one name. Because the store lowercases a locator after encoding it, folding the escape’s hex digits but not the character itself, a name containing an uppercase non-ASCII character needs its ASCII-only fold probed as well. Getting that wrong hides exactly the rows an attacker chose the name to hide.
Review questions: Are SQLite parameters used for all untrusted values? Are purl and CPE normalisations covered by tests? Are metadata backfill paths tested? Are custom overrides scoped by CVE and purl prefix?
Specific to shard stores, and the surface most likely to produce a confidently wrong “clean” result.
Risks: a purl type with no local shard silently resolving to nothing; a stale shard being read as current; a consumer collapsing “no data” and “no findings”; an interrupted refresh leaving a torn database that answers subtly wrong; an implicit network fetch during a scan in an environment that forbids it.
Mitigations: single-purl searches raise ShardCoverageError for an uncovered
type; batch searches mark components coverage_gap: true and attach a coverage
companion; staleness is a separate signal from a gap, never merged into one; an
empty database exits 1 with a message rather than printing no results; full
image refreshes are staged, validated and swapped with the old files stashed
first, so an interruption leaves files missing (loudly) rather than mixed;
shard refreshes use atomic directory renames; on-demand fetch is opt-in through
VDB_AUTO_FETCH.
Review questions: Can this change make an uncovered type look clean? Does it preserve the distinction between a gap and staleness? Would an interrupted version of this operation leave a database that answers rather than one that fails?
Index creation and VACUUM create large transient files. Risks: exhausting
/tmp or runner disk; unexpected WAL or journal files inside published
artifacts; memory-backed temp stores on small machines; index creation changing
final file size unexpectedly.
Controls: VDB_TEMP_DIR, file-backed temp store pragmas, cache size controls,
page size controls, and focused database-size tests.
Review questions: Is VDB_TEMP_DIR set in large workflows? Are temp files
excluded from published artifacts? Does finalization close connections after
index creation and VACUUM? Are sizes measured after finalization?
The CLI downloads pre-built artifacts via ORAS; users may also take Hugging Face files directly. Risks: selecting the wrong variant for a scan; trusting an unexpected registry or dataset URL; stale local databases; unsafe extraction or overwrite; missing provenance in internal mirrors; downloading a compression the unpacker does not handle and then searching an absent database.
Review questions: Does documentation steer production users toward internal mirrors? Are database URLs configurable but never silently changed? Are freshness and metadata checks clear? Is a failed or empty download an error that leaves the previous database intact?
Risks: compromised actions or unpinned dependencies; excessive permissions;
token leakage; publishing partial or stale artifacts; incorrect
NVD_START_YEAR, GITHUB_PAGE_COUNT or VDB_IGNORE_* settings; undocumented
artifact sizes after schema changes; a shard set published without passing its
gate.
Note that build-scope settings are a security control, not only a size control.
A build with an unintended year floor or distro exclusion produces an artifact
that is smaller, valid-looking, and quietly missing advisories. vdb.meta
recording the settings is what makes that detectable after the fact.
Review questions: Are actions pinned where practical? Are tokens printed or
exposed? Are artifact sizes measured from the generated files? Are variant
settings captured in vdb.meta? Did the shard gate pass before the push?
Parameterized SQLite queries on search and insertion paths. Separate data and
index databases so source blobs and search indexes stay distinct. Normalized
source-data storage to remove blob amplification. Temp-store pragmas preferring
file-backed SQLite temp files under VDB_TEMP_DIR. Tests for source
conversion, search behaviour, custom data, pragmas and database-size
regressions. A match-set equivalence gate that proves a storage or performance
change does not alter which vulnerabilities are reported. A store-to-lookup
reachability audit that catches rows written under locators no scan can
propose. Empty-database and coverage-gap guards that turn a silent empty answer
into an explicit failure. Staged, validated, atomically swapped refreshes.
Cached zip support for deterministic local tests without full downloads.
Environment-driven source filtering for app-only and distro-specific builds.
Private GitHub Security Advisory intake for sensitive reports.
Public artifacts depend on upstream feed quality and availability, and feeds are frequently less precise than the vendors they describe. A faithful conversion of an imprecise advisory is still an imprecise answer.
Enumerated version sets remain a known source of false negatives where a feed supplies no range and no statement of intent.
CPE-derived rows discard the CPE components the purl form cannot hold, so an OS
and an application CPE with the same vendor and product collapse into one row.
NVD o: and h: advisories are deliberately not stored, since the distro feeds
own OS packages; a CPE query for an upstream OS product is not bridged to the
distro rows that cover it.
Advisory text in cve_metadata_text is keyed on the CVE id alone, so where
several feeds describe one CVE, every row for it renders a single feed’s text.
This affects displayed text, not matching.
Full app+OS builds remain large and can exhaust weak runners if VDB_TEMP_DIR
and disk capacity are not sized correctly.
Custom data is trusted operator input unless an integration exposes it to less-trusted users.
Schemas are optimized for fresh rebuilds; migration of older databases is not guaranteed and not attempted.
Downstream consumers may treat VDB results as authoritative without checking scope, age or variant. The library reports coverage and staleness, but cannot force a consumer to read them.
Does this change affect source ingestion, storage, search results, version comparison, coverage reporting, downloads or publishing?
Can malformed or very large input cause memory, disk or CPU exhaustion?
Can this change make missing data look like a clean result?
Can a row this change writes still be reached by the lookup meant to find it?
Are secrets, tokens, paths or environment values logged or published?
Are SQL queries parameterized where any external value is involved?
Are row counts, source hash counts and file sizes tested where relevant?
Does the change alter public artifacts or their metadata, and is the size effect measured rather than estimated?
Does documentation explain any new environment variable or operational requirement?
Is a private advisory more appropriate than a public issue for this finding? See SECURITY.md.