Status: implemented as vdb/lib/shard_store.py (client), vdb/lib/shard_split.py
(splitter), with dispatch points in vdb/lib/search.py and the refresh front
end in vdb/lib/db_cmd.py. This document describes the design as built, the
invariants it maintains, and the reasons for the decisions. Measurement history
lives in contrib/shard_store.md; the SQL each step runs is catalogued in
DATABASE.md.
flowchart TB
CLI["vdb CLI"]
DS["dep-scan and other scanners"]
MCP["MCP server"]
EP["search.py entry points<br/>(purl, batch, hydration)"]
UNSAFE["shard-unsafe entry points<br/>(cpe, cve id, text, malware)"]
FULL["canonical full-DB path<br/>(pre-shard code, unchanged)"]
RAISE["PartialDatabaseError"]
FAN["shard_store fan-out<br/>+ coverage"]
FETCH["fetch_shards / auto-fetch"]
MAIN[("main DB pair + vdb.meta")]
STORE[("shard store ($VDB_SHARDS_DIR)")]
REG[("registry: ghcr.io/appthreat/vdb7-*")]
CLI --> EP
DS --> EP
MCP --> EP
CLI -->|"vdb db refresh"| FETCH
EP -->|"completeness: full"| FULL
FULL --> MAIN
EP -->|"completeness: partial<br/>shard-safe"| FAN
EP -->|"completeness: partial<br/>shard-unsafe"| RAISE
UNSAFE --> RAISE
FAN -->|"read-only"| STORE
FAN -->|"partial main participates"| MAIN
FAN -->|"opt-in: VDB_AUTO_FETCH"| FETCH
FETCH -->|"staged + atomic place"| STORE
FETCH --> REG
The full v7 database is a pair of SQLite files, on the order of 15.6 GB uncompressed for the production pair. Workloads scoped to one or two ecosystems (a Python-only scanner, a Debian image scan) pay for the whole feed set. The shard system publishes, alongside the full image, one artifact per purl type plus group shards, so a pypi-only workload holds roughly 90 MB.
Sharding a vulnerability database is not a partitioning problem, it is an epistemic one. A client holding 3 of 43 shards is not a smaller database; it is a database with holes. The engineering constraints follow from that:
Every published artifact, full or shard, carries a vdb.meta JSON manifest
(“meta v2”). The fields the client depends on:
| Field | Meaning |
|---|---|
completeness |
full for complete databases, partial for shards. Drives dispatch. |
artifact.kind |
full or shard; mirrors completeness. |
artifact.name |
Shard name (pypi, deb, app, cpe); identity within the store. |
artifact.types |
Purl types the artifact serves. Authoritative for resolution. |
build_id |
<run_id>-<attempt> of the producing build. Per-shard identity. |
created_utc |
Build time, used for relative staleness. |
compression |
Encoding the artifact was fetched with (xz, zst, none). |
siblings |
registry, tag, available: where sibling artifacts live. |
read_shard_meta() parses a directory’s manifest into an immutable
ShardRecord. Three outcomes, deliberately distinct: None for a directory
that is not a shard (no meta, or a full meta); ShardStoreError for a
directory that claims to be a shard but cannot participate (partial meta
without build_id, or missing either database file); a record otherwise. A
malformed shard is never silently skipped and never silently used.
The type-to-shard mapping is defined once in shard_store.py and imported by
the splitter, so the writer and the reader share one table.
The row space is partitioned by CLAIMED_TYPES (what the group shards take),
with cpe as the complement; the per-type ecosystem shards then overlap that
partition as additional views:
Row-space partition (decided by the splitter via CLAIMED_TYPES):
+---------+---------+---------+---------------------------+
| deb | rpm | apk | app |
| | | | npm pypi maven golang |
| | | | nuget gem composer cargo |
| | | | github |
+---------+---------+---------+---------------------------+
| cpe (complement) |
| generic + every purl type no group shard claims |
+---------------------------------------------------------+
Overlapping views (NOT a partition; both artifacts answer the type):
type npm served by shard npm AND shard app
type julia served by shard julia AND shard cpe
Resolution picks the most specific participant (fewest served types,
then name), so npm resolves to shard npm, not shard app, and a query
runs against the smallest artifact that can answer it.
Group shards cover one family each: deb, rpm, apk, and app, where
app bundles the nine application ecosystems (APP_TYPES: npm, pypi, maven,
golang, nuget, gem, composer, cargo, github). Ecosystem shards (ECOSYSTEM_SHARDS)
map each purl-spec type to itself, minus the types that are already group
shards and minus generic. cpe is the complement shard: every purl-prefix
type not claimed by a group shard (CLAIMED_TYPES), which is where CPE vendor
types and generic land. Its type set is computed at split time and can only
be learned from a shard’s own artifact.types.
Ecosystem shards overlap group shards by design. The npm shard and the
app shard both serve npm; they are separate views, not a partition. This is
why ecosystem shards do not claim their type in CLAIMED_TYPES: removing
julia from cpe would strand it for any store that only carries the group
plus cpe shards.
canonical_shard_for_type() inverts the table statically for fetch decisions
(which shard should serve a type). Whether a local shard does serve a type is
answered by resolution against actual records (section 6).
shard_mode() returns true when the connected main database’s own manifest
declares completeness: partial. The check (search._connected_db_is_partial)
reads the manifest beside the main index file and caches the verdict keyed by
path and mtime, so it costs one file stat per search after the first. Nothing
else gates dispatch: a full main database means the pre-shard code path runs
unchanged, regardless of what is in the store directory.
flowchart TD
E["search entry point"] --> M{"vdb.meta of the<br/>connected main DB"}
M -->|"completeness: full"| FULL["canonical full-DB path<br/>(no dispatch, no coverage object)"]
M -->|"completeness: partial"| SAFE{"entry point<br/>shard-safe?"}
SAFE -->|"search_by_cpe_like, search_by_cve,<br/>metadata/text family, latest_malware"| RAISE["PartialDatabaseError"]
SAFE -->|"search_by_purl_like, search_packages,<br/>get_cve_data(_batched)"| COV["coverage_for_types(requested types)"]
COV -->|"uncovered types and VDB_AUTO_FETCH"| FETCH["fetch_shards(missing)<br/>then re-resolve coverage"]
COV --> RES["resolve_shard_for_type<br/>(specificity order)"]
RES --> CANON["search._search_by_purl_like_on<br/>against the owning shard conns"]
RES -->|"no shard serves the type"| REFUSE["ShardCoverageError (single purl)<br/>or coverage_gap flag (batch)"]
Five dispatch points in search.py route to the fan-out implementations:
Caller in search.py |
Fan-out implementation |
|---|---|
search_by_purl_like |
shard_store.fan_out_purl_like |
search_packages (batch/BOM) |
shard_store.fan_out_search_packages |
get_cve_data (hydration) |
shard_store.fan_out_hydrate_batched |
get_cve_data_batched |
shard_store.fan_out_hydrate_batched |
| BOM summary (coverage attach) | coverage_for_types over locator types |
Four entry points are shard-unsafe and raise PartialDatabaseError on a
partial main database via _raise_unless_full_db: search_by_cpe_like,
search_by_cve, the metadata/text search family, and latest_malware. These
answers are properties of the whole corpus (a CVE id search must sweep every
type; alias and description indexes live per-shard and cannot be unioned
soundly), so a shard genuinely cannot answer them and the error is the honest
result.
Store layout:
$VDB_HOME/ default shard store root: $VDB_SHARDS_DIR
data.vdb7 ... the connected main DB
shards/
pypi/data.vdb7 + data.index.vdb7 + vdb.meta
npm/...
.incoming-pypi-<pid>-<hex>/ fetch staging (never participates)
npm.trash-<hex>/ superseded artifact awaiting deletion
Main-database participation:
main pair ($VDB_HOME) store ($VDB_HOME/shards)
data.vdb7 + data.index.vdb7 pypi/ npm/ deb/
| | | |
+------------------+---------------+----------+---------+
|
participates iff its vdb.meta says completeness: partial,
under its own artifact.name (here: pypi)
discover_store() lists the store root, skips dot-directories and anything
containing .trash-, parses each remaining directory with read_shard_meta,
and keys results by artifact.name. Skipping trash matters for correctness,
not hygiene: a superseded artifact and its replacement share a name, and
directory order would silently decide which participates.
A partial main database participates as a shard under its own artifact.name
(main_shard()). This is what a user gets by downloading a shard artifact
through the ordinary full-image flow: the files land in VDB_HOME and the
main connection becomes that shard’s connection.
participating_shards() returns the discovery result sorted by
(len(types), name) (the specificity order resolution depends on) plus the
common build_id: the single id when the store is uniform, None when
mixed. It raises ShardStoreError only when nothing participates at all.
One collision is a hard error: the connected main database and a store member
claiming the same shard name from different builds raise
BuildIdMismatchError. Both artifacts would compete to answer the same types,
and preferring one by directory order is a silent data downgrade. The message
names both directories and both build ids; the resolution is an operator
action.
resolve_shard_for_type(ptype, records) walks the participating records in
specificity order and returns the first whose artifact.types contains the
type. With both pypi and app present, pypi wins (fewer served types), so
a query runs against the smallest artifact that can answer it. The scoping key
is the type segment of purl_prefix (_prefix_type), the same expression the
splitter uses in SQL (substr(purl_prefix, 5, instr(purl_prefix || '/', '/') - 5)).
The type column is known to disagree with the prefix for some feeds and is
never used for routing.
db.py predates shards and owns a single pair of module-global connections
that every source class captures at construction. Rather than thread shard
state through it, _open_shard_conns() borrows the machinery: it saves
db.db_conn, db.index_conn, db.tables_created, and db.bound_paths,
clears them without closing (the caller’s main connections stay open), calls
db.get(record.db_file, record.index_file, read_only=True) so URI handling,
immutable flags and the vers_compare SQL function registration stay in one
place, steals the resulting pair into the shard module’s cache, and restores
the saved globals in a finally. The invariant, asserted by
test_shard_store.py::test_db_globals_untouched_by_fan_out, is that on exit
db’s globals hold exactly what they held on entry. bound_paths must travel
with the save/restore: leaving a shard’s paths bound would make the caller’s
next explicit get() on its own database look like a binding conflict.
Shard connections are cached in _SHARD_CONNS, keyed by the realpath pair,
behind a double-checked lock. A fan-out over five shards opens five
connections once and reuses them for the process lifetime. reset() closes
them all, clears the attach scratch state and the warning dedup sets; it is
called before any placement (section 11) and by tests and store re-syncs.
sequenceDiagram
participant C as fan-out caller
participant SS as _open_shard_conns
participant DBG as db module globals
participant K as _SHARD_CONNS cache
C->>SS: open(record)
SS->>K: lookup (db_path, index_path)
alt cached
K-->>SS: (data_conn, index_conn)
else cache miss
SS->>DBG: save (db_conn, index_conn, tables_created, bound_paths)
SS->>DBG: clear globals (without closing: caller conns stay open)
SS->>DBG: get(shard db_file, index_file, read_only=True)
Note over DBG: URI handling, immutable flags and<br/>vers_compare registration stay in one place
DBG-->>SS: fresh shard pair
SS->>DBG: restore saved globals (finally)
SS->>K: store pair
end
SS-->>C: (data_conn, index_conn)
Note over DBG: on exit, globals hold exactly what they held on entry
fan_out_purl_likeA purl is strictly type-scoped, so the fan-out for one purl is one shard
query. The sequence: compute the type from the search spec; take a coverage
snapshot for that one type (coverage_for_types); set
search.last_shard_coverage; then either resolve the shard and run the
canonical single-database body search._search_by_purl_like_on against that
shard’s connections, or refuse.
Refusal happens in two cases. An uncovered type raises ShardCoverageError,
because a bare list has no in-band place to say “not checked” and an empty
return would read as “not vulnerable” to every caller that ignores
.coverage. A covered type that fails resolution (coverage and resolution
disagree) also raises: that state means the store changed mid-search, and an
empty list would be a silent under-report.
fan_out_search_packagesThe batch path groups locators by owning shard and answers each group with the
same canonical body as 8.1, so per-shard semantics cannot drift from the
full-database path. Grouping algorithm: one pass over the input assigns each
index to a bucket keyed by shard name, or to __main__ for non-purl locators
(CPE strings, aliases, package names; these run against the main database via
search._search_locator), or to __uncovered__ when the locator’s type is in
coverage.uncovered_types. Gap locators produce a normal per-component
summary with coverage_gap: true and no results. Summaries come back in input
order; the result is a CoverageList carrying the coverage companion.
flowchart TD
IN["input locators (BOM components)"] --> COV["coverage_for_types(all purl types)"]
COV --> G{"assign each locator<br/>to one bucket"}
G -->|"type in uncovered_types"| GAP["__uncovered__ bucket:<br/>summary with coverage_gap: true"]
G -->|"no purl (cpe, alias, package_name)"| MAIN["__main__ bucket:<br/>_search_locator on the main DB"]
G -->|"type covered"| B["bucket keyed by owning shard name"]
B --> SEQ["sequential: canonical body<br/>per shard (default)"]
B --> ATT["attach: UNION ALL index sweep<br/>(index-shaped calls only)"]
GAP --> OUT["summaries in input order,<br/>CoverageList with .coverage"]
MAIN --> OUT
SEQ --> OUT
ATT --> OUT
The ATTACH strategy (section 10) substitutes the union sweep for the
canonical body per locator, but only when the call is index-shaped: no
with_data, no filters, no max_results, no offset. Hydrated or filtered
searches always use the canonical path so hydration semantics cannot drift.
fan_out_hydrate_batchedConsumers such as dep-scan hydrate pass-1 index hits through
get_cve_data_batched with the main connections. In shard mode (the dispatch
fires when index_conn is omitted and the main database is partial) the hits
partition by the same authoritative type segment, each partition resolves to
its owning shard, and hydration runs per shard through the ordinary batched
query. The consumer needs no shard knowledge; hits produced by a fan-out
hydrate against the shards that produced them.
Coverage (dataclass, also serialized via as_dict()) carries:
requested_types frozenset types the caller's locators asked about
covered_types frozenset types a participating shard serves
uncovered_types frozenset requested minus covered; empty means complete
shards_used tuple participating shards serving requested types
build_id str|None common build id, or None for a mixed store
shard_build_ids dict per-shard build identity for shards_used
stale_shards tuple shards lagging their siblings past threshold
It reaches callers two ways: as .coverage on the returned CoverageList
(a plain list subclass, so existing signatures and iteration are unchanged,
and coverage defaults to None so a bare instance can never masquerade as
complete), and on search.last_shard_coverage for callers that receive plain
lists.
Coverage object
|
+---------------+------------+--------------+---------------+
| | | | |
.coverage on search.last_ stderr warning coverage_gap BOM-level
the CoverageList shard_ (once per type per component coverage
(result object) coverage per process) (batch, BOM) section
(opt-in read) (plain lists)
Reporting channels are layered so no caller posture loses the signal.
Uncovered types print one stderr warning per type per process
(_warn_uncovered, suppressed under VDB_QUIET), stating that results for
those types are not checked. Batch summaries carry coverage_gap per
component, which survives stderr suppression and metadata-ignoring callers
both. Single-purl searches raise instead, as described above. BOM-level
summaries attach a store-level coverage section.
The gap decision is derived from the coverage object alone in every path, so disabling coverage computation disables each visible signal at once; a regression test depends on that coupling.
VDB_SHARD_FANOUT selects sequential (default) or attach.
Sequential runs the canonical per-shard body once per participating shard. Its property is provable equivalence: per-shard results are exactly what the full-database path produces for those types, because it is the same code against a smaller database.
sequential (default) attach (VDB_SHARD_FANOUT=attach)
caller caller
| open conns (cached per shard) | one scratch :memory: conn,
v | vers_compare registered on it
shard pypi index: query (round trip) v
| ATTACH pypi/data.index.vdb7 ro
v ATTACH npm/data.index.vdb7 ro
shard npm index: query (round trip) ATTACH deb/data.index.vdb7 ro
| |
v v
shard deb index: query (round trip) ONE statement, UNION ALL over
| every attached cve_index, same
v predicate per branch
merge in Python |
| v
' metadata: routed per owning shard
hydration: per shard (both
strategies attach indexes only)
Attach exists for round-trip latency. _attach_scratch_conn() creates one
in-memory apsw connection, registers vers_compare on it (the sweep’s WHERE
clause calls the function; a connection without it fails), and attaches each
participating shard’s index file read-only under alias shard_<name> with
non-alphanumerics folded to underscores. _attach_purl_sweep() then runs the
index sweep as one statement:
SELECT cve_id, type, namespace, name, vers, purl_prefix
FROM shard_pypi.cve_index
WHERE purl_prefix IN (?, ?, ...) AND vers_compare(?, vers)
UNION ALL
SELECT ... FROM shard_npm.cve_index WHERE ...
Each branch carries the canonical predicate, including the sha-exclusion
pre-clause (search._vers_query) with its arguments per branch. Custom-data
override merging mirrors the canonical path. Metadata cannot ride the sweep
(the scratch connection holds only indexes), so each hit’s metadata is
fetched from the shard owning its type, batches routed per owner through the
ordinary _attach_metadata.
Two limits bound the strategy. SQLite’s SQLITE_MAX_ATTACHED defaults to 10
while a complete store presents roughly 12 candidates, so attach serves the
stores it can and is not universal. Hydration through the scratch connection
would require attaching the data files too, doubling attachments, which is
why hydration is per-shard in both strategies. The strategy is validated by
the same match-set gate as sequential and measured in
contrib/bench.py shard-fanout; sequential won the default on the
equivalence argument, the difference at realistic store sizes being small.
The client layer, db_cmd.refresh_shards, resolves registry, tag and
compression in this priority: explicit arguments, then the siblings record
of any local shard manifest (the store knows where it came from), then
documented defaults (ghcr.io/appthreat, v7.0.x, xz). Two guards run
before anything is downloaded. A tag already ending in -xz or -zst
suppresses suffix appending, so an explicit override cannot become
v7.0.x-zst-xz. A zst resolution with no decompressor available
(zstd_support.available() is None) fails immediately with a message
naming both remedies, because the zst artifact’s layers are bare .zst files
and the download would waste hundreds of megabytes before failing at
unpacking.
_artifact_target() composes <registry>/vdb7-<shard>:<tag>[-<compression>]
per the published naming. The library layer, fetch_shards, performs its own
resolution when the caller passes neither registry nor tag, inheriting the
encoding a local shard was fetched with. db_cmd always passes all three, so
the CLI path resolves in one place; scripts calling fetch_shards directly
with only registry= and tag= suppress its resolution block and must pass
compression= too (documented in MIGRATING_TO_V7).
Each shard fetches into a private staging directory
.incoming-<shard>-<pid>-<hex> under the store root, so concurrent fetchers
never share a staging path. After download, _validate_fetched_shard demands
a parseable shard manifest whose artifact.name equals the shard asked for;
a mismatched artifact is refused rather than placed under the wrong name.
_place_shard implements placement. It calls reset() first: cached
connections into the store must be closed before renames, because POSIX
renames open files silently while Windows raises. Fresh install is a single
os.replace of the staging directory onto the final path, which is atomic;
a concurrent reader observes either no such shard or a complete one. Refresh
renames the current directory aside to <shard>.trash-<hex>, renames the
staging directory in, and deletes the trash; on failure the trash is renamed
back, so the previous artifact survives. Any interruption leaves every shard
either refreshed or at its previous complete artifact, never a mixture, and
the store- discovery rules (section 5) ensure a half-deleted trash directory
cannot shadow its replacement.
On any failure the staging directory is removed and the error is wrapped in
ShardFetchError with context, including the doc 08 §7.4 note that a shard
advertised in siblings.available but missing from the registry may mean the
publisher has not finished publishing the manifest, which is the last thing
pushed.
sequenceDiagram
participant R as fetch_shards / refresh_shards
participant REG as registry (ghcr.io)
participant STG as staging dir (.incoming-...)
participant SLOT as store slot (shards/name)
R->>R: resolve registry, tag, compression<br/>(explicit, else siblings record, else defaults)
R->>R: guards: tag already suffixed, zst needs a decompressor
R->>REG: pull vdb7-shard:tag-enc (resolved reference)
REG-->>STG: layers (.tar.xz unpacked, bare .zst decompressed)
R->>STG: validate: meta is partial, artifact.name matches shard
R->>R: reset() (close cached shard connections)
alt fresh install (slot does not exist)
R->>SLOT: os.replace(staging, slot), atomic
else refresh (slot exists)
R->>SLOT: rename slot aside to slot.trash-hex
R->>SLOT: os.replace(staging, slot)
R->>SLOT: delete trash
Note over SLOT: on rename-in failure the trash is<br/>restored: previous artifact survives
end
Note over STG,SLOT: any failure removes staging, the store<br/>keeps exactly what it had before
VDB_AUTO_FETCH (default off) turns coverage_for_types into the fetch hook.
When requested types resolve to no local shard, the missing shards are
derived via canonical_shard_for_type, fetched through the identical
staged-validated-atomic path, and coverage is re-resolved. A fetch failure
propagates as ShardFetchError; the search does not continue past an
unfulfilled auto-fetch, and the local store is never modified on failure. The
gate exists because a vulnerability scan that silently reaches for the
network is a supply-chain and latency surprise for embedders.
Shards are refreshed independently, and a mixed-build store is a normal state rather than corruption. This follows from constraint 2: since no query combines rows from two shards, build consistency between shards was never load-bearing for matching. Mixed builds change freshness only, which purl types are answered from how recent a dataset.
Freshness is reported, not enforced. stale_shards() computes lag relative
to the newest known build time in the participating set; a shard more than
STALE_SHARD_LAG_DAYS (7.0) behind triggers one stderr warning per
(name, build_id) per process, naming the build ids, ages, and the refresh
command. The threshold is a constant, not a knob: the publish cadence is
daily, so a seven-day lag is roughly seven missed builds of that
ecosystem’s advisories, while shorter lags are ordinary refresh jitter.
The computation requires at least two shards with parseable build times; a
lone shard has no siblings to lag. A shard with an unparseable build time is
never claimed stale, because unknown is not old, though its build id is still
reported. A uniformly old store is the ordinary “data is old” condition
reported by the main database’s own metadata, not a shard-mixing signal.
ShardError is the base. ShardStoreError covers malformed or unreadable
local state. ShardFetchError covers fetch failures (network, registry,
validation, missing oras extra). BuildIdMismatchError covers the
same-name-different-build collision. search.PartialDatabaseError and its
subclass ShardCoverageError cover refusals from the search path. All carry
operator-actionable messages; none surfaces as a bare traceback through the
CLI.
VDB_SHARDS_DIR relocates the store (default $VDB_HOME/shards).
VDB_AUTO_FETCH enables the automatic fetch hook. VDB_SHARD_FANOUT
selects the strategy. VDB_QUIET suppresses the coverage-gap and staleness
warnings (the coverage_gap flag is not suppressed, by design). Fetch-side:
VDB_DATABASE_URL and VDB_APP_ONLY_DATABASE_URL concern the full-image
flow, not shards; VDB_ZSTD_BIN locates the zstd binary for zst artifact
unpacking on Python below 3.14.
Two gates hold the design up. The match-set gate (contrib/match_set.py
shard-gate, run by the publish workflow’s split action) requires, for every
published shard, that shard answers equal full-database answers for the types
the shard serves, over a fixed corpus, prefix probes, absent-type probes and
the shard-unsafe entry points; a shard that fails the gate is not published.
The unit suite pins the structural invariants, including
test_db_globals_untouched_by_fan_out (the connection restore),
TestShardModeDispatch (table identity between splitter and client, at
import time), the placement race tests (mid-deletion directory listings,
trash rollback), and the reference-resolution property tests (no shard
reference is ever left without an encoding suffix).
Why one shard per query rather than merged SQL: it makes per-shard correctness independent of build consistency, lets the canonical search body be reused verbatim, and keeps the proof obligation small; the ATTACH strategy buys back latency where it matters without weakening the contract, because it is gated to the index-only shape and passes the same gate.
Why the type segment of purl_prefix rather than the type column: the
splitter and the client must agree on one scoping key, the prefix segment is
what the splitter’s SQL uses, and the type column is known to disagree with
it for some feeds.
Why coverage rides on the result object: an API that returns “results” cannot add a second required return value without breaking every caller, and a companion attribute plus a module-level mirror lets conservative callers opt in while naive callers still trip the stderr warning or the per-component flag.
Why fetch placement uses rename protocols instead of file copies: a rename is atomic on POSIX for directories within a filesystem, so readers need no locks; the trash-then-replace ordering is the minimal protocol that keeps “reader holds old or new, never mixed” true across an interruption at any point.