Faster alias insert

[crozzy]

No description provided.

[jvdm]

Hey — I tested this on a GKE cluster (StackRox PR stackrox#21503, image tag 4.12.x-363-g01fc4e4b10, SSD storage, dev bundle stream).

The import completes fast (~10 min for all 13 bundles, rhel-vex in 2m17s, zero deadlocks) but the vuln table ends up empty — zero rows. The alias table has 567k entries and last_vuln_update shows all 13 bundles as completed.

I think the issue is that insertAliases consumes vulnIter before the main vulnerability insertion loop runs. vulnIter is a streaming iterator (from jsonblob) that can only be consumed once. After insertAliases exhausts it, the vulnIter(func(vuln *claircore.Vulnerability, ...) bool { loop in the main function sees no data and inserts nothing.

The concurrent test doesn't hit this because it uses slices.Collect(mkVuln(n)) which creates a reusable slice, not a single-pass stream.

Here's the relevant sequence in updateVulnerabilities:

// This consumes the iterator:
if err := s.insertAliases(ctx, vulnIter); err != nil {
    return uuid.Nil, err
}

// ... later, this sees an exhausted iterator:
vulnIter(func(vuln *claircore.Vulnerability, iterErr error) bool {
    // never called — iterator is already consumed
})

Performance-wise the alias insertion approach looks great — just needs the iterator consumption fixed.

[jvdm]

Follow-up: I pushed a fix to jvdm/quay-claircore@fix-faster-alias-insert (commit c43edc9). It removes the separate insertAliases pre-pass and instead collects alias namespaces and aliases during the main vulnIter loop, then bulk-inserts them via a separate connection (outside the transaction) after the loop — preserving the deadlock-avoidance property. It also adds a test using a sync.Once-guarded single-pass iterator to cover this case.

Redeployed on the same GKE cluster (1 replica, premium-rwo SSD, dev bundle, StackRox PR stackrox#21503 image tag 4.12.x-364-g5d7de62ac2):

Bundle	Duration
alpine	13s
aws	21s
debian	20s
epss	30s
manual	<1s
nvd	52s
oracle	1m53s
osv	2m33s
photon	24s
rhel-vex	12m42s
stackrox-rhel-csaf	13s
suse	9m37s
ubuntu	4m45s
Total	~34 min

Metric	Buggy PR (prev comment)	Fixed PR	Baseline (no PR)
vuln rows	0	8,147,014	>5M
alias rows	567,653	548,917	N/A
rhel-vex	2m17s	12m42s	~34 min
Total import	~10 min	~34 min	~55 min
Deadlocks	0	0	0

The buggy run was fast (~10 min) only because it skipped all vuln inserts. With the fix, rhel-vex drops from ~34 min (baseline) to 12m42s and total import from ~55 min to ~34 min.

[jvdm]

Follow-up with 3 matcher replicas (same cluster, premium-rwo SSD, dev bundle, autoscaling disabled):

Per-bundle timing with the work distributed across 3 concurrent pods (each pod processes all bundles, but ON CONFLICT DO NOTHING means only the first to reach a bundle does the real insert):

Pod	Bundles processed	Key durations
6bnfd	alpine, epss, osv, ubuntu	osv 3m, ubuntu 5m
lxwsx	aws, nvd, photon, rhel-vex	rhel-vex 20m56s
mvz6m	debian, oracle, stackrox-rhel-csaf, suse	suse 12m23s, oracle 2m

Wall-clock from first bundle start to last pod "completed update": ~24 min (bottlenecked on lxwsx processing rhel-vex).

Metric	1 replica	3 replicas
vuln rows	8,147,014	8,147,014
alias rows	548,917	548,917
rhel-vex	12m42s	20m56s
Total import	~34 min	~24 min
Deadlocks	0	0
Import errors	0	0

Zero deadlocks with 3 concurrent replicas — the outside-transaction alias namespace/alias insertion is working as intended. rhel-vex is slower per-pod (20m56s vs 12m42s) compared to the 1-replica run, but overall wall-clock improves because suse (12m23s) and rhel-vex (20m56s) run in parallel on different pods.

[jvdm]

crozzy#89

In case you want to incorporate the fixes.