Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/aw/actions-lock.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@
"repo": "github/gh-aw-actions/setup",
"version": "v0.84.3",
"sha": "c863074b673419603d146aab585e2986ef08deec"
},
"github/gh-aw-actions/setup@v0.85.4": {
"repo": "github/gh-aw-actions/setup",
"version": "v0.85.4",
"sha": "2709137ea6c5b0e19aa621454dc643ea8dc526b1"
}
Comment thread
jonathanpeppers marked this conversation as resolved.
}
}
375 changes: 180 additions & 195 deletions .github/workflows/nightly-fix-finder.lock.yml

Large diffs are not rendered by default.

154 changes: 78 additions & 76 deletions .github/workflows/nightly-fix-finder.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,38 +29,60 @@ on:
permissions:
contents: read
issues: read
pull-requests: read
# ###############################################################
# Select a PAT from the pool and override COPILOT_GITHUB_TOKEN.
# Run agentic jobs in an isolated `copilot-pat-pool` environment.
#
# When org-level billing is available, this will be removed.
# See `shared/pat_pool.README.md` for more information.
# ###############################################################
#
# The PAT pool authenticates Copilot requests only. Repository writes use the
# workflow GITHUB_TOKEN, so generated commits and PRs are authored by
# github-actions[bot].
imports:
- uses: shared/pat_pool.md
with:
environment: copilot-pat-pool

environment: copilot-pat-pool
checkout:
- fetch-depth: 0
jobs:
conclusion:
permissions:
issues: write
network:
allowed:
- defaults
- github
- dotnet
safe-outputs:
github-token: ${{ secrets.GITHUB_TOKEN }}
assign-to-agent:
github-token: ${{ secrets.ANDROID_TEAM_PAT }}
model: gpt-5.6-sol
target: "*"
create-issue:
close-older-issues: false
expires: 7d
create-pull-request:
allowed-base-branches:
- main
allowed-files:
- src/**
- tests/**
- Documentation/**
auto-close-issue: false
draft: false
fallback-as-issue: false
labels:
- automated
- code-quality
max-patch-files: 20
title-prefix: "[fix-finder] "
noop: null
missing-data:
create-issue: false
missing-tool:
create-issue: false
noop:
report-as-issue: true
report-incomplete:
create-issue: false
report-failure-as-issue: false
steps:
- env:
Expand Down Expand Up @@ -89,7 +111,7 @@ steps:
bash -o pipefail "$SCRIPT_PATH"
} > /tmp/gh-aw/agent/scan-results.md
echo "✅ Script $SCRIPT_NAME complete → /tmp/gh-aw/agent/scan-results.md"
description: Nightly scan for random code improvement opportunities, files issues assigned to Copilot
description: Nightly scan that implements one random code improvement and opens a PR
model: gpt-5.6-sol
engine:
id: copilot
Expand All @@ -111,38 +133,28 @@ engine:
max-daily-ai-credits: -1
max-ai-credits: -1
strict: true
timeout-minutes: 30
timeout-minutes: 120
tools:
bash:
- find src -name "*.cs" -type f
- find .github/workflows/nightly-fix-finder -name "*.sh"
- grep:*
- wc:*
- head:*
- tail:*
- sort:*
- cat:*
- awk:*
- sed:*
- shuf:*
- date:*
- xargs:*
- basename:*
edit:
bash: ["*"]
github:
github-token: ${{ secrets.GITHUB_TOKEN }}
mode: gh-proxy
min-integrity: none
toolsets:
- repos
- issues
- pull_requests
- search
---

# Nightly Fix Finder

You are the Nightly Fix Finder Agent — an expert system that scans the dotnet/android repository each night for random code improvement opportunities and files actionable issues for Copilot to fix.
You are the Nightly Fix Finder Agent — an expert coding agent that scans the dotnet/android repository each night for a random code improvement opportunity, implements one safe fix, validates it, and opens a PR.

## Mission

Each night, one scan script is selected at random and run. Your job is to read that script's pre-collected output, find one specific actionable improvement, score it against a confidence rubric, and — only if it clears the bar — create a well-scoped issue and assign Copilot to fix it.
Each night, one scan script is selected at random and run. Read that script's pre-collected output, find one specific actionable improvement, score it against the confidence rubric, and — only if it clears the bar — implement and validate the fix in this run, then open one well-scoped PR. Do not create a finding issue.

## Current Context

Expand All @@ -163,12 +175,12 @@ Using the script's guidance and pre-collected sample data, pick **one specific,
1. **Read the actual source file(s)** involved to understand the full context
2. **Verify the issue is real** — not a false positive
3. **Determine the fix** — what specifically needs to change
4. **Scope it appropriately** — one issue should be completable in a single PR
5. **Check for duplicates** — search existing issues for similar titles before proceeding
4. **Scope it appropriately** — the complete fix should fit in one small PR
5. **Check for duplicates** — search open issues and PRs for the same problem before proceeding

### Phase 2.5: TFM / Language-Version Sanity Check (MANDATORY)

Before writing any code into the issue's `Suggested Fix`, locate the **owning `*.csproj`** for the file you intend to change (walk up parent directories until you find one) and read its `<TargetFramework>` / `<TargetFrameworks>` and `<LangVersion>` values. The emitted code MUST compile against every TFM in that list. The following APIs have non-obvious version floors and are the most common compile-break sources:
Before changing code, locate the **owning `*.csproj`** for the file you intend to change (walk up parent directories until you find one) and read its `<TargetFramework>` / `<TargetFrameworks>` and `<LangVersion>` values. The implementation MUST compile against every TFM in that list. The following APIs have non-obvious version floors and are the most common compile-break sources:

| API / syntax | Minimum TFM / LangVersion | Safe fallback for older TFMs |
|---|---|---|
Expand All @@ -187,96 +199,86 @@ This step exists because PR #11455 emitted `ArgumentNullException.ThrowIfNull` i

## Phase 3: Score Against Confidence Rubric

Before filing, score the proposed fix on a 0–30 scale across three dimensions. Be honest — under-scoring is far cheaper than filing a bad issue.
Before changing files, score the proposed fix on a 0–30 scale across three dimensions. Be honest — under-scoring is far cheaper than opening a bad PR.

| Dimension | 0 | 5 | 10 |
|---|---|---|---|
| **Actionability** — can Copilot implement this from the issue alone? | Vague / requires design discussion | Clear intent but missing concrete code change | Specific file + lines + exact replacement code |
| **Actionability** — can you implement and validate this now? | Vague / requires design discussion | Clear intent but uncertain implementation or validation | Specific file + lines + exact change and targeted validation |
| **Safety** — what is the blast radius if the fix is wrong? | Behavior change to shipped public API, native code, or runtime | Touches MSBuild task logic or non-trivial managed code | Purely additive, comment-only, test-only, or fully covered by existing tests |
| **Scope** — is this completable in a single small PR? | Sprawls across many files or requires deep refactor | Multiple files but cohesive | One file, single hunk, ≤30 lines changed |

**Threshold: ≥ 22 / 30 to file.** Additionally, **safety must be ≥ 6** — any fix scoring lower on safety must be declined regardless of total. The SkiaSharp project that pioneered this rubric confirmed it correctly stops risky behavior-change fixes that otherwise look attractive.

If the proposal scores below either bar, call `noop` with a message that includes the score breakdown and why you declined.

## Phase 4: Create Issue
**Threshold: ≥ 22 / 30 to implement.** Additionally, **safety must be ≥ 6** — any fix scoring lower on safety must be declined regardless of total. The SkiaSharp project that pioneered this rubric confirmed it correctly stops risky behavior-change fixes that otherwise look attractive.

Create exactly **one** well-scoped issue using `create_issue`. The issue must be specific enough that Copilot can implement the fix without ambiguity.
If the proposal scores below either bar, call `noop` with a message that includes the score breakdown and why you declined. Do not modify files.

### Issue Template
## Phase 4: Implement and Validate

Use this structure:
Implement the fix yourself:

```markdown
### Problem
1. Make the smallest complete change that resolves the verified problem.
2. Follow all repository instructions and existing style. Never modify generated files, non-English localization files, or unrelated code.
3. Add or update a focused test when behavior changes or a regression test is practical.
4. Run the smallest targeted build or test command that covers the changed behavior. A PR requires successful validation; if the fix cannot be validated in this environment, revert only your own changes and call `noop`.
5. Review `git diff` for accidental or unrelated edits.
6. Commit the final changes with a concise message ending in:

[1-2 sentences describing what's wrong and why it matters]
`Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>`

### Location
The workflow configures Git as `github-actions[bot]`, so keep that author identity. Do not override Git author or committer settings.

- **File(s)**: `path/to/file.cs`
- **Line(s)**: [specific lines if applicable]
## Phase 5: Open the PR

### Current Code
After the fix is committed, call `create_pull_request` exactly once. Use a short branch name and a PR body with this structure:

[Show the relevant code snippet]
```markdown
> AI-generated fix. Produced by the `nightly-fix-finder` agentic workflow.

### Suggested Fix
### Problem

[Describe exactly what should change, with example code if possible]
[What was wrong and why it mattered]

### Guidelines
### Fix

- [Any repo-specific conventions to follow]
- [Reference to relevant documentation]
[What changed, including the key files]

### Acceptance Criteria
### Validation

- [ ] [Specific, verifiable criteria]
- [ ] All tests pass
- [ ] No new warnings introduced
[Exact build/test commands run and their results]

### Fix-finder metadata

- Script: `<script-name>` (e.g. `04-missing-xml-docs`)
- Script: `<script-name>`
- Score: `<n>/30` (actionability: `a`, safety: `s`, scope: `c`)
```

## Phase 5: Assign to Copilot

After creating the issue, use `assign_to_agent` to assign Copilot to work on it. You **MUST** pass the `issue_number` parameter — use the `temporary_id` from the `create_issue` call (**without** the `#` prefix). The safe-output is configured with `model: "claude-opus-4.8"` so Copilot will use Claude Opus 4.8 to implement the fix.

Example call sequence:
1. `create_issue` with `temporary_id: "aw_fix123"`, `title`, `body`
2. `assign_to_agent` with `issue_number: "aw_fix123"`

## Rules

1. **One issue per run** — Create exactly one issue, not multiple
2. **Be specific** — The issue must be implementable from the description alone
3. **Verify before filing** — Read the actual source to confirm the issue is real
4. **Honor the confidence gate** — Below 22/30 or safety <6 ⇒ `noop`, not "file anyway"
5. **Skip non-actionable findings** — If the selected script's data is empty or all false positives ⇒ `noop`
6. **Respect repo conventions** — Follow dotnet/android formatting and coding style
7. **Don't duplicate** — Search for existing issues with similar titles before creating
1. **One PR per run** — Open exactly one review-ready PR, never multiple
2. **No issues** — Never call an issue-creation tool or use an issue as a fallback
3. **Implement completely** — Do not open a PR containing only analysis, a TODO, or a partial fix
4. **Verify before changing** — Read the actual source and confirm the problem is real
5. **Validate before opening** — Do not open a PR unless the targeted validation passes
6. **Honor the confidence gate** — Below 22/30 or safety <6 ⇒ `noop`, not "fix anyway"
7. **Skip non-actionable findings** — Empty scan data, false positives, duplicates, or changes outside the allowed paths ⇒ `noop`
8. **Respect repo conventions** — Follow dotnet/android formatting, testing, localization, and coding rules

## Adding a New Category

The fix-finder is intentionally easy to extend:

1. Drop a new `NN-name.sh` file into `.github/workflows/nightly-fix-finder/`
2. Add the script name (without `.sh`) to the `workflow_dispatch` → `script` → `options` list at the top of this file so it appears in the GitHub Actions UI dropdown
3. Print a `GUIDANCE` heredoc first (what to look for / how to fix / what NOT to flag)
3. Print a `GUIDANCE` heredoc first (what to look for / how to implement / what NOT to change)
4. Print `## Scan Data` followed by your grep/find output
5. Run `gh aw compile` to regenerate `nightly-fix-finder.lock.yml`

The nightly `shuf` picks up the new script automatically; updating the dropdown is only needed for manual dispatch.

## Important

You **MUST** end by calling exactly one set of safe output tools:
You **MUST** end by calling exactly one safe output tool:

- **`create_issue` + `assign_to_agent`**: When a valid improvement clears the confidence gate
- **`create_pull_request`**: After a valid improvement clears the gate, is fully implemented, committed, and validated
- **`noop`**: When no actionable improvement was found, or the proposal scored below the gate

```json
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ scan excludes `Tests/`, `*Test.cs`, `*Tests.cs`).

### How to fix
Either implement the TODO or remove the comment if it's no longer relevant.
Include a brief explanation in the issue body of why it was resolved or removed.
Include a brief explanation in the PR body of why it was resolved or removed.

### What NOT to flag
- TODOs inside generated files (`*.generated.cs`, `*.Designer.cs`) — never touch those
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ follow clear patterns.
**Scope: shipped code only.** The scan deliberately excludes test
infrastructure (`/Tests/`, `/Test/`, `*Test.cs`, `*Tests.cs`) because we don't
spend cycles enabling NRT in non-shipped code. If you can't find a shipped-code
candidate in the scan data, call `noop` — do NOT file an issue for a test file.
candidate in the scan data, call `noop` — do NOT open a PR for a test file.

### ⚠️ CRITICAL — Check the owning csproj TargetFramework FIRST

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ test fields) by declaring the field as nullable.
- Logical-not operators (`!foo`, `!IsEnabled`, `if (!x)`) — only the
*postfix* null-forgiving form (`foo!.`, `foo![`, `foo!;`, `foo!,`, `foo!)`)
- Strings/comments containing `!` (the scan filters trailing-`!` positions
so this is rare, but still verify before filing). Note: the regex
so this is rare, but still verify before changing it). Note: the regex
intentionally does NOT exclude the `!=` operator at the line level --
the character class after `!` already excludes `=`, so excluding lines
that contain `!=` anywhere would drop legitimate hits.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ hold the `#region` directive.
- Generated files (`*.generated.cs`, `*.Designer.cs`, `*.g.cs`)
- Files under `external/` (submodules — not owned by this repo)
- `#region` appearing inside a string literal or comment (rare but verify
by opening the file before filing)
by opening the file before changing it)
GUIDANCE

echo ""
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/nightly-fix-finder/05-general-mistakes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ Read the randomly selected source files thoroughly and look for **real bugs**:
- Incorrect exception handling (catching too broadly, swallowing without logging)

### How to fix
File a specific issue describing the actual bug with concrete evidence (line
numbers, the suspected wrong behavior, and the expected correct behavior).
Implement the smallest complete fix for the actual bug and add a focused
regression test when practical.

### What NOT to flag
- Formatting, whitespace, or style — not actionable for a fix issue
- Formatting, whitespace, or style — not actionable for a fix PR
- "Could be cleaner" subjective preferences with no functional impact
- Generated files (`*.generated.cs`, `*.Designer.cs`, `AssemblyInfo.cs`)
GUIDANCE
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ cat << 'GUIDANCE'
### What to look for
Files in `src/` with >10 `using` directives that likely contain unused ones.
A high count alone isn't a bug — the agent MUST open the file and confirm
specific directives are unused before filing.
specific directives are unused before changing it.

### How to fix
Remove the unused `using` directives. Keep `global using`s and conditional
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ entry exists yet, that means a new `XA####` code is needed in
- Test code under `Tests/` / `tests/`.
- The `AsyncTask` base class itself, and the `AsyncTaskExtensions` file.
- The agent MUST open the file and verify the enclosing class actually
derives (directly or transitively) from `AsyncTask` before filing.
derives (directly or transitively) from `AsyncTask` before changing it.
GUIDANCE

echo ""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,10 @@ Hardcoded error/warning strings passed to `Log.Error`, `Log.Warning`,
should live in `Properties.Resources` for localization.

### How to fix
1. Add the new message to the English `src/.../Properties/Resources.resx`
(never modify non-English `*.resx` or `*.lcl` files — those are auto-generated)
2. Reference it via `Properties.Resources.XA####`
3. If you create a NEW `XA####` error code, you MUST also:
- Create `Documentation/docs-mobile/messages/xa####.md` following the
existing format (frontmatter + Example messages + Issue explanation + Solution)
- Add the new code to the table of contents in
`Documentation/docs-mobile/messages/index.md`
Replace the literal with a suitable existing `Properties.Resources.XA####`
message. If no existing resource accurately describes the error, call `noop`:
adding a new resource requires updating the generated `Resources.Designer.cs`,
which this workflow must not modify.

### What NOT to flag
- Strings already coming from `Properties.Resources`
Expand Down
Loading