perf: optimize combine-to-osv bucket listing and avoid partial files.txt upload#5632
Open
another-rex wants to merge 1 commit into
Open
perf: optimize combine-to-osv bucket listing and avoid partial files.txt upload#5632another-rex wants to merge 1 commit into
another-rex wants to merge 1 commit into
Conversation
…txt upload - Added upload of files.txt list file to cve5 and nvd-cve-osv converters. - Optimized combine-to-osv to read files.txt instead of listing the GCS bucket. - Added fallback to bucket listing if files.txt is too old or fails to read. - Made list file max age configurable via flag. - Skipped files.txt upload on partial runs (start-year > default). - Refactored converters to use channels instead of global mutexes for tracking output files.
another-rex
force-pushed
the
optimize-osv-bucket-listing
branch
from
ff4bc52 to
8ea47ce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR optimizes the bucket listing operation in
combine-to-osvby using a pre-uploadedfiles.txtlist file generated by thecve5andnvd-cve-osvconverters.files.txtlist file tocve5andnvd-cve-osvconverters.combine-to-osvto readfiles.txtinstead of listing the GCS bucket.files.txtis too old or fails to read.files.txtupload on partial runs (start-year > default).agy