Benchmarks

Trustcheck publishes a reproducible fixed-input performance and correctness comparison of trustcheck scan against the workflow-pinned pip-audit 2.10.1. The benchmark uses a versioned corpus manifest and the OSV advisory service for both tools. Only those two command paths contribute timing or correctness samples. The Trustcheck command explicitly uses --fast, limiting it to advisory lookup and lockfile/requirements parsing for an apples-to-apples comparison.

The corpus is benchmarks/corpus/corpus.json. Version 2026.06 contains 135 package entries: 100 mixed clean and historically vulnerable PyPI pins, marker and extra cases, private-index inputs, pip-tools, PEP 751, uv, Poetry, and PDM lockfiles, VCS and editable dependencies, intentionally malformed requirements, and dedicated resolution/profile evidence cases. Direct timing and correctness use only cases marked compare_with_pip_audit.

Comparable requirements cases audit their declared pins directly. Trustcheck and pip-audit both use --no-deps, and pip-audit also uses --disable-pip, so historical releases do not execute build backends or need to resolve into one compatible environment. The README table is therefore a fixed-input --no-deps comparison, not a full dependency-resolution benchmark.

Correctness is alias-aware: advisories match when any normalized CVE, GHSA, PYSEC, or provider ID overlaps. The raw unmatched records remain in the JSON result so agreement cannot hide feed or normalization differences. Recall is measured against the signed benchmarks/corpus/truth.json, which contains independently curated vulnerable and clean package-version pairs, aliases, fixed versions, withdrawals, markers, extras, and private-index cases. truth.json.sig is verified with the checked-in public key before a benchmark runs. The manifest's minimum recall and maximum false-positive gates make the benchmark exit nonzero on a Trustcheck regression.

Separate evidence suites publish complete dependency resolution and Trustcheck standard/full profile results. Every suite records cold-cache and warm-cache p50/p95, peak RSS, request samples where the tool reports them, and exact commands. Resolution evidence compares complete package/version sets. Advisory recall uses the signed curated truth corpus as its reference. Profile evidence records how many artifacts had provenance, verification, static inspection, native analysis, and heuristic findings. Trustcheck request counts come from report diagnostics; pip-audit request counts are null because the tool does not expose that measurement.

The benchmark workflow runs manually, after the release workflow completes, and on a weekly schedule. It publishes the raw JSON as a retained workflow artifact and prints the generated README evidence table into the workflow summary for maintainer review. Publication requires at least five warm samples per tool, a signed truth corpus with declared correctness gates, no truth-corpus regressions, and no one-sided advisory findings. Local runs default to benchmarks/results/latest.json; commit or publish that file only when it was regenerated from the current corpus and environment. pip-audit exits 1 when it finds vulnerabilities; trustcheck scan exits 0 here because the benchmark intentionally uses its default non-blocking policy. Commands that return an accepted exit code but no output are retried twice; the result records this setting and includes retry time in the timing sample.

python benchmarks/benchmark_against_pip_audit.py

Wall-time results include package metadata and advisory requests plus output generation. The benchmark reports observed performance rather than claiming feature-equivalent work.

Malicious-package calibration

The pip-audit benchmark does not calibrate malicious-package scoring because it runs trustcheck scan --fast and excludes artifact, history, AST, native binary, and dynamic-analysis heuristics. The seed manifest for that separate security milestone is benchmarks/corpus/malicious-calibration.json.

That manifest is versioned and defines required strata for known malicious PyPI releases, typo-squats, benign native-extension packages, legitimate packages that use powerful capabilities, and deliberately weird but harmless academic or development packages. It is currently marked seed-unmeasured; until reviewed entries and a reproducible runner are published, malicious-package confidence and false-positive values remain estimated rule priors rather than empirical measurements.

Acceptance matrix

The Acceptance Matrix workflow is separate from fast pull-request and push CI. It runs nightly and manually across Linux, macOS, Windows, and every supported Python version. Each job executes one real trustcheck scan case from scripts/acceptance_matrix.py and uploads the rendered report.

The matrix covers selected public packages and corpus fixtures for pip-tools, uv, Poetry, PDM, PEP 751 pylock.toml, extras and markers, private-index directives, native wheels, and sdists. The private-index fixture accepts the expected upstream failure for intentionally internal package names but still requires a rendered report.