feat: detect AI agent invocations in telemetry#9127
Conversation
github-actions
Bot
added
pr/external
stage/needs-triage
reedham-aws
left a comment
reedham-aws
left a comment
There was a problem hiding this comment.
I'm approving because I think it works as is, but I'd like to also hear your answers to the questions I've left in comments. Overall, great work!
| Whether it is running in the standalone GitHub Copilot CLI (not Copilot-in-VS-Code | ||
| or the cloud agent), via COPILOT_AGENT_SESSION_ID (a per-session UUID from the | ||
| @github/copilot bundle; undocumented, so re-verify per release) or | ||
| GITHUB_COPILOT_CLI_MODE=="true". It must NEVER key on a CI variable: Copilot's | ||
| cloud agent runs inside GitHub Actions, which cicd.py already detects via | ||
| GITHUB_ACTION, so any GITHUB_* rule would false-match ordinary CI. |
There was a problem hiding this comment.
nit: I find this comment a little confusing, mostly because it's a little jargony. It's not clear to me what we should "re-verify" and what the importance of the CI variable is.
There was a problem hiding this comment.
Good catch. I think it would be worth it to rewrite the doc string so that it does not have a lot of confusing function names and key words. We should also drop the GITHUB_COPILOT_CLI_MODE as it is not needed anymore.
| cloud agent runs inside GitHub Actions, which cicd.py already detects via | ||
| GITHUB_ACTION, so any GITHUB_* rule would false-match ordinary CI. | ||
| """ | ||
| return "COPILOT_AGENT_SESSION_ID" in environ or environ.get("GITHUB_COPILOT_CLI_MODE", "") == "true" |
There was a problem hiding this comment.
Where does the GITHUB_COPILOT_CLI_MODE envvar come from? I couldn't find it in any documentation.
There was a problem hiding this comment.
The GITHUB_COPILOT_CLI_MODE does exist, but the CLI does not assign it into a spawned process's env, so it wouldn't be present when Copilot invokes SAM. Hence, it would not be useful for us so i'll go ahead and delete it and only detect on COPILOT_AGENT_SESSION_ID
If you'd like you can see more about GITHUB_COPILOT_CLI_MODE in the published bundle: github link.
| # exact name, not a prefix, so GEMINI_CLI_HOME / GEMINI_CLI_NO_RELAUNCH don't match | ||
| Agent.GeminiCLI: "GEMINI_CLI", | ||
| Agent.Kiro: _is_kiro, | ||
| # OpenCode sets OPENCODE=1 in its root CLI middleware |
There was a problem hiding this comment.
I'm curious where you found this, I only see it when directly examining the code, but couldn't find it documented anywhere.
There was a problem hiding this comment.
Good catch, it's not documented anywhere official I also found it by directly reading OpenCode's source. It's set in the root CLI middleware here: github link
| toolkit_user_agent = os.environ.get(USER_AGENT_ENV_VAR, "").strip() | ||
| if toolkit_user_agent and ACCEPTED_USER_AGENT_FORMAT.match(toolkit_user_agent): | ||
| return toolkit_user_agent |
There was a problem hiding this comment.
nit: I think we can leave this as user_agent for the variable name, just because it's not necessarily guaranteed to to be the toolkit.
| @patch.dict( | ||
| "samcli.lib.telemetry.agent_detector.os.environ", | ||
| {"GITHUB_ACTION": "run1"}, | ||
| clear=True, | ||
| ) | ||
| def test_detector_does_not_identify_github_actions_as_copilot(self): | ||
| # Plain GitHub Actions CI (no Copilot session var) must never be attributed to | ||
| # GitHub Copilot; agent detection returns None so cicd.py can own GITHUB_ACTION. | ||
| self.assertIsNone(AgentDetector().agent()) |
There was a problem hiding this comment.
Should this also set the COPILOT_AGENT_SESSION_ID to confirm that it gets ignored?
Add an AgentDetector, modeled on the existing CICDDetector, that
identifies when SAM CLI is invoked by an AI coding agent by reading
environment variables the agent sets, and records it via the existing
userAgent telemetry field (emitted as "<agent>/1.0"). This lets us
distinguish agent-driven usage from human and CI/CD usage.
Detected agents and their signals:
- Claude Code CLAUDECODE present
- Codex any CODEX_* var present
- Cursor CURSOR_AGENT present
- Gemini CLI GEMINI_CLI present
- Kiro TERM_PROGRAM == kiro, or AWS_EXECUTION_ENV contains
amazonq/kiro
- OpenCode OPENCODE present
- GitHub Copilot COPILOT_AGENT_SESSION_ID present, or
GITHUB_COPILOT_CLI_MODE == "true"
Shared variables (AWS_EXECUTION_ENV, TERM_PROGRAM) are substring or
exact-value matched, never presence-checked, to avoid misattributing
Lambda/CloudShell/CodeBuild or generic VS Code environments as agents.
GitHub Copilot detection deliberately avoids any GITHUB_* / GITHUB_ACTION
key so it never false-matches ordinary GitHub Actions CI, which the
existing CICDDetector already handles.
An existing AWS_TOOLING_USER_AGENT (AWS toolkit) takes precedence over a
detected agent. Amazon Q is intentionally deferred; until it is added
after Kiro in resolution order, Amazon Q CLI runs resolve to kiro.
Includes unit tests for each agent, the exact-name/too-generic guards
(GEMINI_CLI, OPENCODE vs AGENT), the AWS_EXECUTION_ENV and GITHUB_ACTION
false-positive guards, and toolkit-takes-precedence behavior.
1e1f9c1
2303620 to
1e1f9c1
Compare
madhavdonthula1
left a comment
madhavdonthula1
left a comment
There was a problem hiding this comment.
New minor changes that removed:
GITHUB_COPILOT_CLI_MODE == "true"- test cases
3 participants
Add an AgentDetector that identifies when SAM CLI is invoked by an AI coding agent by reading environment variables the agent sets. Records it via the existing userAgent telemetry field (emitted as "/1.0"). This lets us distinguish agent-driven usage from human and CI/CD usage.
Detected agents and their environmental variable signals:
Shared variables (AWS_EXECUTION_ENV, TERM_PROGRAM) are substring or exact-value matched, never presence-checked, to avoid misattributing Lambda/CloudShell/CodeBuild or generic VS Code environments as agents. GitHub Copilot detection deliberately avoids any GITHUB_* / GITHUB_ACTION key so it never false-matches ordinary GitHub Actions CI, which the existing CICDDetector already handles.
An existing AWS_TOOLING_USER_AGENT (AWS toolkit) takes precedence over a detected agent. Amazon Q is intentionally deferred; until it is added after Kiro in resolution order, Amazon Q CLI runs resolve to kiro.
Includes unit tests for each agent, the exact-name/too-generic guards (GEMINI_CLI, OPENCODE vs AGENT), the AWS_EXECUTION_ENV and GITHUB_ACTION false-positive guards, and toolkit-takes-precedence behavior.