Skip to content

fix(analyzer): bound file analysis workers - #8058

Merged
cx-artur-ribeiro merged 3 commits into
Checkmarx:masterfrom
omribz156:codex/analyzer-bounded-workers
Sep 7, 2026
Merged

fix(analyzer): bound file analysis workers#8058
cx-artur-ribeiro merged 3 commits into
Checkmarx:masterfrom
omribz156:codex/analyzer-bounded-workers

Conversation

@omribz156

Copy link
Copy Markdown
Contributor

Closes #8046

Reason for Proposed Changes

  • The analyzer currently starts one goroutine per candidate file.
  • On very large repositories, that can create tens of thousands of goroutines during "Preparing Scan Assets" and trigger the thread-exhaustion crash reported in the issue.

Proposed Changes

  • Replace per-file goroutine spawning with a bounded worker pool.
  • Size the pool from GOMAXPROCS, capped at 128 workers and never larger than the candidate file count.
  • Add unit coverage for the worker-count helper.

Verification

  • go test ./pkg/analyzer -count=1
  • go test ./pkg/scan -run "Test.*Analyze|Test.*Init|Test.*Prepare" -count=1
  • git diff --check

This was implemented with Codex assistance, with the final patch kept focused and manually reviewed.

I submit this contribution under the Apache-2.0 license.

@devAL3X

devAL3X commented May 27, 2026

Copy link
Copy Markdown

I have tested this fix against my repo where the issue was occurring - the issue is no longer reproducible

@cx-artur-ribeiro

cx-artur-ribeiro commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Hi @omribz156,
Thanks for the contribution 🙏 !

I've also verified this resolves the thread-exhaustion crash cleanly and the bounded-pool wiring (channel-fed workers, wg.Add/wg.Done moved to the pool level) is correct, including preserving the existing unwanted-channel drain race handling in computeValues that I've introduced earlier this year.

One addition I would propose: the added test to analyzer_test only covers analyzerWorkerCount as a pure function. Nothing exercises Analyze() itself under a file count that forces multiple files through a single pooled worker, so a regression that reintroduced go a.worker(...) per file would pass this test suite untouched.
I suggest adding the test bellow alongside it, in pkg/analyzer_test.go.

Let me know what you think and thanks again for your help improving kics!
Thank you as well @devAL3X for exposing the problem and taking the time to explain what the problem was as well.

P.S: I can address the changes on my side and close this PR once we merge the changes, mentioning both the issue and the original pull request.

Comment thread pkg/analyzer/analyzer_test.go
@cx-artur-ribeiro

Copy link
Copy Markdown
Contributor

Commits do need to be signed in order for any pull request to be merged.

If this is something that you cannot achieve on your side, I will open the PR on my side and tag you, as well as the pull request and related issue as well.

Let me know if you could sign the commits on your side @omribz156.

@cx-rui-araujo

Copy link
Copy Markdown
Contributor

Hi @omribz156,
Thanks for the contribution 🙏 !

The bounded pool implementation looks good.
I have one suggestion: could we make the 128 worker cap configurable via a MaxAnalyzerWorkers field in the Analyzer struct? This would let users who consume the package adjust it for their specific environments.
I added comments with the suggestion in case you wanna commit via GitHub.
P.S: I can address these changes on my side in a different PR after merging this one (so you can resolve my comments if you feel so).

Let me know what you think and thanks again for your help improving KICS!
Also, as Artur said, thank you as well @devAL3X for exposing the problem and taking the time to explain what the problem was as well.

Comment thread pkg/analyzer/analyzer.go
Comment thread pkg/analyzer/analyzer.go Outdated
Comment thread pkg/analyzer/analyzer.go Outdated
@cx-artur-ribeiro
cx-artur-ribeiro force-pushed the codex/analyzer-bounded-workers branch from 2880735 to 8724dcd Compare September 7, 2026 14:07

@cx-rui-araujo cx-rui-araujo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐐

@cx-artur-ribeiro
cx-artur-ribeiro merged commit f118059 into Checkmarx:master Sep 7, 2026
5 of 35 checks passed
cx-andre-pereira pushed a commit that referenced this pull request Sep 8, 2026
* fix(analyzer): bound file analysis workers

* test: cover bounded analyzer worker concurrency

* update: add MaxAnalyzerWorkers to Analyzer providing better usage for library embedders

---------

Co-authored-by: Artur Ribeiro <153724638+cx-artur-ribeiro@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(core): thread exhaustion when scanning large repositories

4 participants