🏥 Repo Health Dashboard — dotnet/machinelearning

[github-actions]

Last updated: 2026-06-20 07:14 UTC | Run: #27863907066 | Status: ⚠️ Warning

📊 Executive Summary

Repository health is ⚠️ Warning — marginal improvement today. Stale PR finding resolved: #7609 (arm64 SymSgd) received community activity today and is no longer stale; dependency-bot PRs #7607 and #7570 are actively maintained by the bot (both updated within 30 days). Core warning findings remain stable: 100+ untriaged issues (Day 10) and 7-issue blocking-clean-ci cluster (Day 3) with #6978 seeing another active update today. No new critical findings. AzDO pipeline monitoring disabled — no AZDO_PAT configured.

---

🔴 Critical Findings

None

---

🟡 Warning Findings

ID	Finding	Status	Age
I3-untriaged	100+ open issues with untriaged label (prior full count: 221+)	📌 EXISTING	Day 10
I2-no-milestone	821+ open issues without milestone — triage/planning gap	📌 EXISTING	Day 10
C5-blocking-ci-6978	#6978 blocking-clean-ci: TorchSharp QA test OOM/killed — updated today 07:00 UTC	📌 EXISTING	Day 5
C6-blocking-ci-cluster	7 open blocking-clean-ci issues: #6978, #7067, #7343, #7393, #7417, #7418, #7419	📌 EXISTING	Day 3

---

✅ Recently Resolved

ID	Finding	Resolved
P5-stale-prs	Stale non-baselined PRs cleared: #7609 received community activity today; bot PRs #7607 and #7570 both updated within 30 days	2026-06-20

---

📋 Baselined Findings

ID	Finding	Baselined Since
I1-P0-5805	#5805 P0: MKLImports PDB missing — known long-term tech debt	2026-06-10
I-P1-aged	23+ P1 issues (stable long-tail backlog)	2026-06-10
P5-baselined-prs	PRs #7416, #7406, #6749, #6664 — feature/4.0 branch or long-running WIP	2026-06-11

---

📈 Trends (7-day)

Metric	Value	Change
P0 issues	1	→ stable
P1 issues	23+	→ stable
Open bugs	60+	→ stable
Open PRs (visible)	15+	→ stable
Stale PRs (non-baselined, >30d)	0	✅ ↓ from 3
blocking-clean-ci issues	7	→ stable
Locker workflow	✅ success	→ healthy
Backport workflow	✅ success (ran 2026-06-19 14:02 UTC)	→ healthy
AzDO pipelines	⚫ N/A	No AZDO_PAT

---

🔄 Active PRs Under Review

PR	Title	Age	Status
#7626	Fix memory leak & stale-model in PredictionEnginePool	6d	In review
#7625	Add SentencePieceTokenizer factory methods for Unigram	10d	api-ready-for-review
#7609	Enable SymSgdNative on arm64 via system BLAS	46d	↺ Activity today

---

Baseline: .github/health-baseline.md · Workflow run #27863907066

Note

🔒 Integrity filter active — some items filtered by GitHub integrity policy (same as prior runs).

Note

🔒 Integrity filter blocked 214 items

The following items were blocked because they don't meet the GitHub integrity level.

• Error: "Microsoft.ML currently supports 'x64' and 'x86' processor" when target runtime is *linux-arm* or *linux-arm64* #6588 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Exposing the tree for multiclass classification #6370 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook? #6353 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add substitutes for 4 IntelMKL methods so SymSgd can be added to non x86/x64 builds #5798 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Memory leak  #5744 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Migrate to VSTest for all Unit Tests #5587 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML #5569 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SVMLightLoader Fails above 128 dense rows #5566 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• TensorFlow scoring sample should pass an empty dataview #4210 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• mlnet's generated projects don't include the cs files #4145 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Troubles with CustomMappingEstimator after save/load because InputSchemaDefinition is not being saved. #3988 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does GetFeatureWeights support categorical splits? #3766 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• How to inspect OneVersusAll models #3701 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [AutoML] Allow to use serialized IDataView as an input #3684 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OvaModelParameters is not strongly-typed #2467 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Conditions in KeyToVectorMappingEstimator.GetOutputSchema seem wrong #2185 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 198 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health Check — Orchestrator · ● 488.9K · ◷

[github-actions]

📋 Daily Delta — 2026-06-17T07:40 UTC

Run: #27673479083

	Count
🆕 New findings	0
📌 Existing findings	4
✅ Resolved	0
📋 Baselined	1 (P0 + 24 P1 + 6 PRs)

Key Changes Since 2026-06-16

• C5-blocking-ci-6978 — Issue #6978 had activity today at 07:00 UTC (still open, still blocking-clean-ci). TorchSharp QA OOM/kill pattern continues to affect Helix CI.
• All other findings are unchanged from yesterday.

CI Status

• backport.yml ✅ — last run 2026-06-16, success
• locker.yml ✅ — healthy (410+ runs, recent all success)
• AzDO pipelines: ⚠️ no AZDO_PAT — skipped

Dispatches This Run

• C5-blocking-ci-6978 → repo-health-investigate (severity: high)
• I3-untriaged → repo-health-investigate (severity: medium)

⚠️ Partial collection: GitHub MCP server failed mid-scan. Untriaged label count and PR stale details carried forward from yesterday's run.

Note

🔒 Integrity filter blocked 64 items

The following items were blocked because they don't meet the GitHub integrity level.

• Error: "Microsoft.ML currently supports 'x64' and 'x86' processor" when target runtime is *linux-arm* or *linux-arm64* #6588 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Exposing the tree for multiclass classification #6370 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook? #6353 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add substitutes for 4 IntelMKL methods so SymSgd can be added to non x86/x64 builds #5798 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Memory leak  #5744 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Migrate to VSTest for all Unit Tests #5587 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML #5569 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SVMLightLoader Fails above 128 dense rows #5566 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• TensorFlow scoring sample should pass an empty dataview #4210 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• mlnet's generated projects don't include the cs files #4145 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Troubles with CustomMappingEstimator after save/load because InputSchemaDefinition is not being saved. #3988 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does GetFeatureWeights support categorical splits? #3766 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• How to inspect OneVersusAll models #3701 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [AutoML] Allow to use serialized IDataView as an input #3684 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OvaModelParameters is not strongly-typed #2467 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Conditions in KeyToVectorMappingEstimator.GetOutputSchema seem wrong #2185 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 48 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health Check — Orchestrator · ● 3.4M · ◷

[github-actions]

🔍 Investigation: C5-blocking-ci-6978

Finding: Issue #6978 — TestSimpleQA killed (exit 137 / OOM) in Helix CI
Severity: High
Category: Issue

---

Evidence

1. Test is already skipped — [Fact(Skip = "Skip in CI build, uses too much memory.")] was added in PR Working on memory issue during tests for TorchSharp #7022 (merged 2024-03-19). The test is not currently running in CI. — Source: test/Microsoft.ML.TorchSharp.Tests/QATests.cs line 29

2. Root memory issue documented — On a local Windows run, TestSimpleQA uses 4.67 GB peak (after cumulative test execution starting from ~750 MB for first test). On Ubuntu 18.04 Helix CI runners, the OOM kill occurs at a lower threshold. — Source: issue Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 comment by @ericstj (2024-01-31)

3. Helix CI runners confirm exit 137 — Original logs show Starting test: TestSimpleQA → Killed → _commandExitCode=137. Pattern is consistent: always killed specifically when TestSimpleQA runs, not intermittent across different tests. — Source: issue Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 body

4. PR Working on memory issue during tests for TorchSharp #7022 was a partial fix — Added Dispose() calls to TorchSharp NasBert and AutoFormerV2 model modules to reduce memory growth between tests. Also skipped TestSimpleQA. Issue was never closed after this workaround. — Source: commit c69c4a0 message and file diffs

5. Issue stuck open with stale labels — Issue has labels blocking-clean-ci, bug, untriaged, and Known Build Error — all from January 2024. No assignee, no milestone. No new comments since 2024-01-31. The "updated today" (2026-06-17T07:00:59Z) triggered by this repo health workflow scanning the label, not new activity. — Source: issue Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 metadata

6. No related PRs open — Search found no open PRs targeting this issue. The TorchSharp update PR update torchsharp to 0.105.0 #7416 (baselined/stale) is the only related pending TorchSharp work. — Source: PR search

---

Root Cause

Confidence: High

The TestSimpleQA test loads a full RoBERTa-based QA model (NasBert architecture) which is extremely memory-intensive. When run sequentially after other TorchSharp tests in the same process, cumulative memory consumption reaches ~4.67–5.5 GB, exceeding the memory limit of Ubuntu 18.04 Helix CI runners (likely 4 GB). The OOM killer issues SIGKILL (exit 137).

The immediate blocker was resolved in March 2024 by skipping the test. The test is not running in CI today. The issue was never closed, leaving stale blocking-clean-ci and untriaged labels that continue to trigger repo-health alerts 2.5 years later.

---

Recommendations

Priority	Action	Who
🔴 Immediate	Close issue #6978 — the CI-blocking symptom is resolved (test is skipped). Remove blocking-clean-ci label first.	@michaelgsharp / @ericstj
🟡 Short-term	Re-enable TestSimpleQA with an isolation strategy — run it in its own dotnet test invocation (separate process) with environment limit enforced, OR add [CollectionDefinition(DisableParallelization=true)] + force GC between tests. Helix supports per-work-item isolation.	@michaelgsharp
i️ Long-term	Reduce QA model memory footprint — investigate whether the RoBERTa/NasBert model can be freed more aggressively after Fit(), or whether a smaller test fixture (fewer layers / fewer epochs) would keep peak memory under 2 GB. Tracking issue: memory leak noted in #5744.	TorchSharp team

---

Related

• Issue: #6978 — original report
• PR: #7022 — partial fix (Dispose + skip)
• Issue: #5744 — memory leak (baselined P1)
• Test file: test/Microsoft.ML.TorchSharp.Tests/QATests.cs

---

Investigated by repo-health-investigate • Finding: C5-blocking-ci-6978

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #7295 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #6976 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health — Investigate Finding · ● 361.5K · ◷

[github-actions]

🔍 Investigation: I3-untriaged

Finding: 221+ open issues with untriaged label — triage process gap persisting for 7+ days
Severity: Medium
Category: Issue

---

Evidence

1. 221+ open issues carry untriaged — stable at this level for 7+ days per dashboard trend (count unchanged between 2026-06-10 and 2026-06-17). — Source: dashboard 🏥 Repo Health Dashboard — dotnet/machinelearning #7634

2. untriaged label policy is milestone-driven — .github/policies/untriaged.yml adds the label on issue open/reopen when no milestone is assigned; removes it only when the issue is closed or assigned to a milestone. There is no time-based or activity-based removal. — Source: .github/policies/untriaged.yml

3. I3-untriaged and I2-no-milestone are the same problem — the 221+ untriaged issues are a direct subset of the 821+ issues without a milestone. Adding any issue to a milestone automatically removes untriaged. No milestone assignment = untriaged forever. — Source: policy logic + dashboard

4. Issue triage agent adds but never removes untriaged — .github/workflows/issue-triage.agent.md (triggered on issues: [opened]) classifies new issues and always adds untriaged alongside other labels. The prompt explicitly says "Always add untriaged alongside any other label." Removal depends entirely on a human assigning a milestone. — Source: .github/workflows/issue-triage.agent.md

5. Oldest visible untriaged issue: ~3 years old — Object detection: Enable loading from checkpoint  #6670 created 2023-05-10 (enhancement + needs-further-triage), never assigned a milestone. — Source: list_issues query (ASC by created_at)

6. Known Build Error (Helix CI bot) issues inflate count — Issues like Failure to load lib_lightgbm in Microsoft.ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2 #6841, ArgumentOutOfRangeException when running Microsoft.ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2 #6842, Microsoft.ML.Tests.TrainerEstimators.TrainerEstimators.TestEstimatorMulticlassNaiveBayesTrainer failed with assert #7060, AutoML tests writing test output outside of a test which causes test process to fail with no failing test cases #7067 are auto-generated by Helix CI (ericstj bot), carry untriaged + blocking-clean-ci + Known Build Error, and are tracked in a separate AzDO system. They were bulk-touched today at 07:00 UTC but still carry untriaged because no milestone is assigned. — Source: list_issues (updated_at DESC)

7. Automated agentic workflow failure reports also get untriaged — [aw] CI Failure Scanner - Feedback (machinelearning) failed #7633 (CI Failure Scanner failure, created today, auto-expires 2026-06-24) carries untriaged. These transient operational issues go through a different lifecycle but are counted in the backlog. — Source: list_issues

8. No closed/resolved untriaged issues in 2026 visible — A search for label:untriaged is:closed closed:>2026-01-01 returned only agentic workflow failure reports (operational, not community issues). Zero community bug/feature untriaged issues have been triaged closed this year. — Source: search_issues

9. Active feature work is happening on untriaged issues — [API] Public way to construct SentencePieceTokenizer (Unigram) from tokenizer.json / in-memory pieces+scores #7624 ([API] Public way to construct SentencePieceTokenizer, created 2026-06-10) has an assignee (Copilot) and 6 comments, yet still carries untriaged simply because no milestone was assigned. This shows the label is losing signal value. — Source: search_issues

---

Root Cause

Confidence: High

The untriaged label is mechanically equivalent to "no milestone assigned." The policy is working as designed — but no systematic milestone assignment process exists, so the label accumulates indefinitely. Three compounding factors worsen the backlog: (1) the oldest community issues have been awaiting triage for 2–3 years, (2) Helix CI and agentic workflow bots auto-create issues that immediately receive untriaged but are managed through separate systems and never get milestones, and (3) the issue-triage.agent.md intentionally preserves untriaged because removing it requires a human milestone decision that never happens. This is a process gap, not a tooling failure.

---

Recommendations

Priority	Action	Who
🔴 Immediate	Bulk-assign Known Build Error + blocking-clean-ci issues to a "Backlog" or "Tracking" milestone (or close as "won't fix" if stale). These are CI-bot-generated and managed in AzDO — they should not count as untriaged community issues. Approx 15–30 issues.	Team
🔴 Immediate	Exclude agentic workflow failure issues ([aw] title prefix or agentic-workflows label) from the untriaged.yml policy — they have expires markers and don't need standard triage.	Team
🟡 Short-term	Create a "Backlog" milestone and assign all enhancement/bug issues that have been reviewed (e.g., by the triage agent) but not yet scheduled. This will deflate the untriaged count while preserving planning flexibility.	Team
🟡 Short-term	Run a triage sprint targeting issues >90 days old with untriaged — close stale/duplicate ones, assign the rest to a milestone. Focus on pre-2024 issues first (~50+ visible, more filtered by integrity).	Team
i️ Long-term	Update issue-triage.agent.md to remove untriaged when it determines an issue is a Known Build Error (has Helix CI structure) or is an agentic workflow failure.	Team
i️ Long-term	Add a stale bot rule: issues with untriaged older than 180 days without activity should be auto-closed or auto-assigned to "Backlog" milestone — this prevents indefinite accumulation.	Team

---

Related

• Dashboard: 🏥 Repo Health Dashboard — dotnet/machinelearning #7634 (I2-no-milestone is the same underlying gap — 821+ open issues without milestone)
• Triage policy: .github/policies/untriaged.yml
• Triage agent: .github/workflows/issue-triage.agent.md
• Sample old untriaged issues: #6670 (2023-05-10), #6719 (2023-06-01), #6735 (2023-06-14)
• CI bot untriaged issues: #6841, #6842, #7060, #7067
• Active work on untriaged issue: #7624 (has assignee but still untriaged)

---

Investigated by repo-health-investigate • Finding: I3-untriaged

Note

🔒 Integrity filter blocked 92 items

The following items were blocked because they don't meet the GitHub integrity level.

• ResNet101 [ShapeInferenceError] Attribute pads has incorrect size #6580 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [Object Detection] Bounding box rotated due to image exif data #6592 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Improvements when specifying a ModelLoader when adding a predictionenginepool to DI. #6618 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does ML.NET support KNN? (part 2) #6639 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: Distributes learning / potential "hack" using checkpoints CSV on a network drive? #6648 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: How to get trainer used for best model #6651 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: Trying to serializing params from TrialResult fails #6652 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: Get columnInformation from binary IDV file #6653 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Need to find multiple columns for the same model for prediction #6680 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: setting temporary folder model  #6687 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Visualization and analysis of LightGBM and FastTree #6688 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Support Multiple Threads on ML Related DB Access  #6692 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Parallelism: trouble saturating CPU when making predictions #6702 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML 2.0: Should Seed be a tunable param? #6704 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Clarifications Regarding ForeCastBySsa #6720 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• AutoML Forecasting Experiment Programaticaly #6725 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 76 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health — Investigate Finding · ● 1.1M · ◷

[github-actions]

📋 Daily Delta — 2026-06-18T07:32 UTC

Run: #27743911734

	Count
🆕 New findings	1
📌 Existing findings	4
✅ Resolved	0
📋 Baselined	1 (P0 + 24 P1 + 6 PRs)

Key Changes Since 2026-06-17

• C6-blocking-ci-cluster 🆕 NEW — Full picture of CI-blocking issues now visible: 7 open blocking-clean-ci issues ( Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978, AutoML tests writing test output outside of a test which causes test process to fail with no failing test cases #7067, SdcaLogisticRegression failing with LogLoss value above 0.5 on Apple M1 #7343, Resource contention in tests DownloadImageSet #7393, Code coverage build fails to instrument file in use #7417, AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async fails in CI #7418, TestAppendMany_SizeMoreThanMaxBufferCapacity triggers OOM killer #7419). All are tracked Known Build Errors and received automated batch status-check updates at ~07:00–07:02 UTC today. Prior runs only tracked Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 individually.
• C5-blocking-ci-6978 — Issue #6978 remains active (Day 3).
• Investigation failure — Issue #7635 was created 2026-06-17 indicating a prior investigation dispatch failed.
• New issue opened: #7636 [ci-scan-feedback] ci-scan: add Hard Rule 10 (created 2026-06-18 04:15 UTC, untriaged).
• New PR activity: #7626 "Fix memory leak hot-swapping PredictionEnginePool" active and updated today; #7625 Draft SentencePieceTokenizer API (Copilot) updated today.

Overall Status

🟡 Warning — unchanged. No new critical findings. The CI blocking issues cluster warrants investigation to understand scope and remediation plan.

Note

🔒 Integrity filter blocked 137 items

The following items were blocked because they don't meet the GitHub integrity level.

• Error: "Microsoft.ML currently supports 'x64' and 'x86' processor" when target runtime is *linux-arm* or *linux-arm64* #6588 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Exposing the tree for multiclass classification #6370 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook? #6353 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add substitutes for 4 IntelMKL methods so SymSgd can be added to non x86/x64 builds #5798 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Memory leak  #5744 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Migrate to VSTest for all Unit Tests #5587 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML #5569 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SVMLightLoader Fails above 128 dense rows #5566 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• TensorFlow scoring sample should pass an empty dataview #4210 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• mlnet's generated projects don't include the cs files #4145 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Troubles with CustomMappingEstimator after save/load because InputSchemaDefinition is not being saved. #3988 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does GetFeatureWeights support categorical splits? #3766 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• How to inspect OneVersusAll models #3701 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [AutoML] Allow to use serialized IDataView as an input #3684 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OvaModelParameters is not strongly-typed #2467 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Conditions in KeyToVectorMappingEstimator.GetOutputSchema seem wrong #2185 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 121 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health Check — Orchestrator · ● 1.2M · ◷

[github-actions]

🔍 Investigation: C6-blocking-ci-cluster

Finding: 7 open blocking-clean-ci issues — all Known Build Errors, all active as of 2026-06-18T07:00 UTC
Severity: Medium
Category: Pipeline / Test Infrastructure

---

Evidence

1. #6978 — TestSimpleQA killed with exit 137 (OOM); TorchSharp QA test accumulates 4.6 GB memory at run time after prior tests consume 1.3 GB. Memory does not fully release between tests. No assignee; no milestone; open since Jan 2024. Hit count: 0/0/0 (sporadic, not recently triggering per tracker).
2. #7067 — AutoMLExperiment logs to TestOutputHelper from a CancellationToken callback that fires after the test completes, crashing the test host with "There is no currently active test." Assignee: LittleLittleCloud; no milestone; open since Mar 2024. Hit count: 0/0/0.
3. #7343 — SdcaLogisticRegression produces LogLoss ~0.52 on Apple M1 (ARM64), exceeding the (0, 0.5) assertion range. Numerical precision divergence between x86 and ARM64 SIMD paths. Assignee: @michaelgsharp; milestone: ML.NET 5.0. Hit count: 0/0/4 — actively firing.
4. #7393 — Concurrent tests race to download flower_photos_tiny_set_for_unit_tests.zip to the same path, causing file-in-use errors. Design discussion underway: recommended fix is grouping image-dependent tests to run serially. Assignee: @LittleLittleCloud; milestone: ML.NET 5.0. Hit count: 0/0/0 (periodic).
5. #7417 — Code coverage instrumentation (Coverlet) fails to lock a .pdb file (Microsoft.ML.TestFramework.pdb) already held by another test process on Windows. Pattern: Unable to instrument module ... The process cannot access the file. No assignee; milestone: ML.NET 5.0. Hit count: 0/1/2 — most actively firing issue in the cluster.
6. #7418 — AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async fails a timing assertion (expected ≤ 2500 ms, actual ~3236 ms). Timing-sensitive test that flakes under CI load. Assignee: LittleLittleCloud; milestone: ML.NET 5.0. Hit count: 0/0/0.
7. #7419 — TestAppendMany_SizeMoreThanMaxBufferCapacity triggers OOM killer (exit 137) in Linux Helix agents, allocating very large DataFrames. No assignee; milestone: ML.NET 5.0. Hit count: 0/0/0.

---

Root Cause

Confidence: High

The 7 issues decompose into three distinct root causes:

Root Cause	Issues
Memory exhaustion (OOM / exit 137) — Tests allocate too much heap for Helix CI agent limits; no explicit GC/dispose between test runs	#6978, #7419
Parallel test isolation failures — Shared resources (download paths, PDB files for coverage) are not protected against concurrent access; async callbacks escape test scope	#7393, #7417, #7067
Platform numeric divergence (ARM64) — SIMD/FP results differ on Apple M1 vs x86-64; test thresholds were calibrated only on x86	#7343

The cluster's persistence (oldest issue Jan 2024) indicates accepted-but-unresolved flakes. The two actively firing issues today are #7417 (Coverlet PDB lock, 1 hit in 7 days) and #7343 (SDCA ARM64, 4 hits in 30 days).

---

Recommendations

Priority	Action	Who
🔴 Immediate	#7417: Add retry/back-off in Coverlet instrumentation, or mark the conflicting test project ExcludeFromCodeCoverage as interim workaround. Most actively firing issue (2 CI hits/month).	@ericstj / eng-services
🔴 Immediate	#7343: Widen SdcaLogisticRegression LogLoss assertion to (0, 0.55) or use platform-specific tolerance to stop blocking ARM64 PRs.	@michaelgsharp
🟡 Short-term	#7393: Use per-test unique download directory or a file-lock guard in DownloadImageSet. Preferred: group image tests in one [Collection("ImageTests")] for sequential execution.	@LittleLittleCloud
🟡 Short-term	#7067: Guard the AutoML CancellationToken logging callback — catch InvalidOperationException before calling QueueTestOutput after test completion.	@LittleLittleCloud
🟡 Short-term	#7418: Increase timing budget to 5000 ms or use a mock timer to remove CI-load sensitivity.	@LittleLittleCloud
i️ Long-term	#6978, #7419: Profile TorchSharp and DataFrame tests; add explicit GC.Collect() + Dispose() between tests, or split memory-intensive tests into isolated Helix work items with larger memory quotas.	@michaelgsharp / @LittleLittleCloud
i️ Long-term	Add a repo-level policy: tests using shared file resources must use [Collection] or thread-safe lazy-init download helpers to prevent recurrence of #7393/#7417 class of issues.	Team

---

Related

• 5 of 7 issues are in ML.NET 5.0 milestone and need resolution before release: SdcaLogisticRegression failing with LogLoss value above 0.5 on Apple M1 #7343, Resource contention in tests DownloadImageSet #7393, Code coverage build fails to instrument file in use #7417, AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async fails in CI #7418, TestAppendMany_SizeMoreThanMaxBufferCapacity triggers OOM killer #7419
• Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 and AutoML tests writing test output outside of a test which causes test process to fail with no failing test cases #7067 are unassigned/unmilestoned and need triage
• Previous finding C5-blocking-ci-6978 is superseded by this cluster investigation

---

Investigated by repo-health-investigate • Finding: C6-blocking-ci-cluster

Generated by Repo Health — Investigate Finding · ● 262.4K · ◷

[github-actions]

🏥 Daily Health Delta — 2026-06-19

	Count
🆕 New findings	0
📌 Existing findings	5
✅ Resolved findings	0
📋 Baselined	1

Changes since 2026-06-18

• C6-blocking-ci-cluster: 🆕 NEW → 📌 EXISTING (Day 2). Seven blocking-clean-ci issues remain open with no resolutions detected.
• C5-blocking-ci-6978: #6978 received an update today at 07:01 UTC — active investigation is ongoing (Day 4).
• P5-stale-prs: Expanded — 3 non-baselined stale PRs now visible (was 1): added #7607 (arcade update, 53d) and #7570 (release/5.0 arcade update, 152d).

🔔 Notable

• #6978 (blocking-clean-ci, TorchSharp QA OOM/exit 137) was updated this morning at 07:01 UTC — someone is actively investigating.
• Two healthy PRs in active review: #7625 (SentencePieceTokenizer) and #7626 (PredictionEnginePool hot-swap fix).
• AzDO pipeline monitoring disabled — AZDO_PAT not configured.

Workflow run #27812569167

Note

🔒 Integrity filter blocked 139 items

The following items were blocked because they don't meet the GitHub integrity level.

• Error: "Microsoft.ML currently supports 'x64' and 'x86' processor" when target runtime is *linux-arm* or *linux-arm64* #6588 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Exposing the tree for multiclass classification #6370 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook? #6353 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add substitutes for 4 IntelMKL methods so SymSgd can be added to non x86/x64 builds #5798 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Memory leak  #5744 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Migrate to VSTest for all Unit Tests #5587 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML #5569 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SVMLightLoader Fails above 128 dense rows #5566 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• TensorFlow scoring sample should pass an empty dataview #4210 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• mlnet's generated projects don't include the cs files #4145 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Troubles with CustomMappingEstimator after save/load because InputSchemaDefinition is not being saved. #3988 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does GetFeatureWeights support categorical splits? #3766 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• How to inspect OneVersusAll models #3701 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [AutoML] Allow to use serialized IDataView as an input #3684 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OvaModelParameters is not strongly-typed #2467 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Conditions in KeyToVectorMappingEstimator.GetOutputSchema seem wrong #2185 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 123 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health Check — Orchestrator · ● 2M · ◷

[github-actions]

🔍 Investigation: C6-blocking-ci-cluster

Finding: 7 open blocking-clean-ci issues — no resolutions in 2 days; #6978 actively updated today
Severity: High
Category: Pipeline / Test reliability

---

Evidence

1. Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978 — TestSimpleQA killed with exit code 137 (OOM) on Ubuntu. Memory progressively builds across TorchSharp test suite, reaching 4.6–5.5 GB by the time TestSimpleQA runs. No assignee. Age: ~2.5 years (Jan 2024). Updated today 07:01 UTC by bot (known-issue heartbeat).
2. AutoML tests writing test output outside of a test which causes test process to fail with no failing test cases #7067 — AutoML CancellationTokenStopTrainingManager fires a log callback after the xUnit test has ended, triggering InvalidOperationException: There is no currently active test. Assigned to @LittleLittleCloud. Age: ~15 months.
3. SdcaLogisticRegression failing with LogLoss value above 0.5 on Apple M1 #7343 — SdcaLogisticRegression produces LogLoss 0.52 on Apple M1 (expected < 0.5). ARM64 floating-point divergence from x86. Assigned to @michaelgsharp. Milestone: ML.NET 5.0. 4 hits in last month — most active of the cluster.
4. Resource contention in tests DownloadImageSet #7393 — Multiple AutoML tests race to download flower_photos_tiny_set_for_unit_tests.zip to the same path, triggering a file-in-use error. Design discussion reached (isolate per-test download folder or serialize downloads). Assigned to @LittleLittleCloud. Age: ~4 months.
5. Code coverage build fails to instrument file in use #7417 — Code coverage instrumenter (coverlet) cannot open Microsoft.ML.TestFramework.pdb because a test process holds it. Windows x64 only. 2 hits in last month. No assignee. Linked to recent PRs [release/5.0] Update dependencies from dotnet/arcade #7570, Add report-green AzDO pipeline so Build Analysis completes on build-excluded PRs #7623.
6. AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async fails in CI #7418 — Timing assertion BeLessOrEqualTo(2500ms) exceeded (~3236ms, +30%) in AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async. Slow CI agents cause flap. Assigned to @LittleLittleCloud. Age: ~3 months.
7. TestAppendMany_SizeMoreThanMaxBufferCapacity triggers OOM killer #7419 — TestAppendMany_SizeMoreThanMaxBufferCapacity in Microsoft.Data.Analysis.Tests killed with exit code 137 (OOM) on Linux. Age: ~3 months. No assignee.

---

Root Cause

Confidence: High

The 7 issues fall into 4 distinct root-cause patterns — this is not a single systemic failure but an accumulated backlog of known, deferred test-reliability bugs:

Pattern	Issues	Description
🧠 Memory / OOM	#6978, #7419	Tests exceed Linux CI memory limits; TorchSharp leaks across test run; Data.Analysis allocates large buffers
📁 File contention	#7393, #7417	Parallel tests share mutable file-system resources without synchronization
⏱️ Timing assertions	#7418	Hard-coded millisecond thresholds don't account for CI agent variability
🍎 ARM64 precision	#7343	SDCA numerical result diverges on Apple M1; threshold not platform-adjusted

The cluster grew from 2 issues (2024) to 7 because individual bugs are filed but not prioritized to fix — the Known Build Error label suppresses CI noise while leaving root causes open indefinitely.

---

Recommendations

Priority	Action	Owner
🔴 Immediate	Fix #7343: relax or make platform-conditional the 0.5 LogLoss threshold for ARM64 — it has 4 active CI hits/month and a clear 1-line fix	@michaelgsharp
🔴 Immediate	Fix #7418: increase the 2500ms timing ceiling to ~5000ms or use a relative multiplier — trivial change with high CI noise reduction	@LittleLittleCloud
🟡 Short-term	Fix #7393: isolate per-test download directory (or serialize DownloadImageSet calls in AutoML tests) — design already agreed in comments	@LittleLittleCloud
🟡 Short-term	Fix #7417: investigate coverlet / test runner PDB file lock ordering on Windows — consider disabling coverage for Microsoft.ML.TestFramework or using shadow-copy	Unassigned → needs owner
🟡 Short-term	Fix #7067: ensure AutoML cancellation callbacks do not capture xUnit ITestOutputHelper after the test has completed (use a null-safe wrapper or unsubscribe in Dispose)	@LittleLittleCloud
i️ Long-term	Fix #6978 + #7419: profile TorchSharp memory leak across test run; consider running TestSimpleQA in isolation or after explicit GC/dispose cycle between tests	Unassigned
i️ Long-term	Establish a policy: new blocking-clean-ci issues require an assigned owner and a fix-by milestone before the Known Build Error suppression is applied — prevents unbounded accumulation	Team process

Related

• Issues: #6978, #7067, #7343, #7393, #7417, #7418, #7419
• 3 of 7 issues have no assignee ( Microsoft.ML.TorchSharp.Tests.QATests.TestSimpleQA followed by process killed / return 137 #6978, Code coverage build fails to instrument file in use #7417, TestAppendMany_SizeMoreThanMaxBufferCapacity triggers OOM killer #7419)
• 4 of 7 are on Milestone ML.NET 5.0 (SdcaLogisticRegression failing with LogLoss value above 0.5 on Apple M1 #7343, Resource contention in tests DownloadImageSet #7393, Code coverage build fails to instrument file in use #7417, AutoMLExperiment_return_current_best_trial_when_ct_is_canceled_with_trial_completed_Async fails in CI #7418)
• Most-active hits: SdcaLogisticRegression failing with LogLoss value above 0.5 on Apple M1 #7343 (4/month), Code coverage build fails to instrument file in use #7417 (2/month)

---

Investigated by repo-health-investigate • Finding: C6-blocking-ci-cluster

Generated by Repo Health — Investigate Finding · ● 195K · ◷

[github-actions]

🔄 Daily delta — 2026-06-20 07:14 UTC

Metric	Yesterday	Today	Change
🆕 New findings	0	0	—
✅ Resolved findings	0	1	+1
📌 Existing findings	5	4	-1
🔴 Critical	0	0	—
🟡 Warning	5	4	-1

✅ Resolved today

• P5-stale-prs — #7609 (arm64 SymSgd) received community activity today (updated_at = now); dependency-bot PRs #7607 and #7570 are within 30-day activity window (bot-maintained)

🔍 Still watching

• #6978 blocking-clean-ci updated again today at 07:00 UTC (Day 5 since flagged) — TorchSharp QA test OOM
• 7-issue blocking-clean-ci cluster remains open (Day 3) — no resolutions since first flagged
• #7625 SentencePieceTokenizer PR at api-ready-for-review (10d) — awaiting API review
• #7626 PredictionEnginePool fix (6d) — in review

i️ No dispatches today

All warning findings are EXISTING (not NEW). Budget saved for new critical/warning findings on next run.

Run #27863907066 · AzDO monitoring offline (no AZDO_PAT)

Note

🔒 Integrity filter blocked 214 items

The following items were blocked because they don't meet the GitHub integrity level.

• Error: "Microsoft.ML currently supports 'x64' and 'x86' processor" when target runtime is *linux-arm* or *linux-arm64* #6588 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Exposing the tree for multiclass classification #6370 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• CreateEnumerable code fails in Visual Studio FSI but works in .NET Interactive notebook? #6353 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Add substitutes for 4 IntelMKL methods so SymSgd can be added to non x86/x64 builds #5798 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Memory leak  #5744 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Migrate to VSTest for all Unit Tests #5587 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OMP: Error #15: Initializing libiomp5.dylib in Image Classification with AutoML #5569 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• SVMLightLoader Fails above 128 dense rows #5566 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• TensorFlow scoring sample should pass an empty dataview #4210 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• mlnet's generated projects don't include the cs files #4145 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Troubles with CustomMappingEstimator after save/load because InputSchemaDefinition is not being saved. #3988 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Does GetFeatureWeights support categorical splits? #3766 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• How to inspect OneVersusAll models #3701 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [AutoML] Allow to use serialized IDataView as an input #3684 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• OvaModelParameters is not strongly-typed #2467 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• Conditions in KeyToVectorMappingEstimator.GetOutputSchema seem wrong #2185 list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• ... and 198 more items

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Repo Health Check — Orchestrator · ● 488.9K · ◷