Add AI disclosure to PR form - #1634
Conversation
Also, tiny wording change for "Expected Usage" section comment. It's tricky to comment out a section; much easier to delete a section. So, now it says to delete it (if it doesn't apply) instead of to comment it out.
Sevans711
added
documentation
| <!-- If you did not use AI, please write "AI Usage: N/A" and remove these checklist items or mark as [N/A]. | ||
| Otherwise, please briefly specify all tools used and how they were used. Include model if known. Examples: | ||
| "AI Usage: Claude (Fable 5) made all code edits, but I came up with the ideas and design for this feature myself." | ||
| "AI Usage: discussion with Gemini and ChatGPT (5.5 Instant) to help find and understand the cause of this bug." |
There was a problem hiding this comment.
Do we care if a developer discussed an issue with an AI if they wound up writing the code themselves?
There was a problem hiding this comment.
Personally, yes, but I'm also curious to hear what others think about this.
My understanding is that using AI influences our thoughts, behaviors, work, etc, even if we do not utilize its suggestions directly. E.g., this study (with faster-to-read press release) finds that being exposed to inline autocomplete suggestions from a biased AI affects people's views, even for people who are informed about the AI's bias ahead of time.
I would guess there are probably similar studies about how your problem-solving gets affected by discussing something with an AI, though I haven't taken a close look at any papers like that yet so I could easily be wrong.
There was a problem hiding this comment.
I would say no, from my experience. Having a back-and-forth with Claude is a very different workflow than just vibing it. And anyway, trying to control that would get us into the position of having to be the thought police.
There was a problem hiding this comment.
I agree with @Sevans711, and this distinction may matter less with time, but I think it's still good to know for now. Without knowing much about user or the model, results can be pretty unpredictable. Like Chatgpt with no access to the whole file, being prompted by a new user may have wildly different credibility than a core user/developer using Claude Code which has access to a variety of inputs, like git, other files, dependencies, and more.
I could also see myself ticking "yes" even if I wrote the code, because I'll use models as "proofreading" after writing code, and it can either expand on or validate my findings. Similar to spell check in writing products. In this case it may be nice to know that the user at least went through some form of proofreading/testing before submitting.
There was a problem hiding this comment.
I guess I just don't see the ROI here. Can we imagine a situation where someone uses AI to think through a problem, writes the code themselves, takes responsibility for the code, and then... what? I can't imagine anything seriously negative happening in that situation.
There was a problem hiding this comment.
How about making the "how" optional, but still there for those who want to share.
There was a problem hiding this comment.
How about making the "how" optional, but still there for those who want to share.
Example phrasing, maybe this would look something like:
## AI Disclosure
<!-- Please specify all AI tools used. Optionally, include model and/or briefly describe usage. Examples:
"AI Usage: Claude (Fable 5), Gemini"
"AI Usage: ChatGPT (5.5 Instant) to help understand cause of this bug, but I wrote all updates myself."
"AI Usage: just GitHub Copilot's inline code suggestions."
If you did not use AI, please write "AI Usage: N/A" and remove these checklist items or mark as [N/A].-->
I would be happy enough with something like that, especially if it sounds good to @rljacob and @cmdupuis3? It avoids requiring a description, while still having the upside of encouraging/allowing one.
There was a problem hiding this comment.
Per concerns around it would be discouraging, how about going lighter-weight with this first implementation (i.e. not ask detailed usage questions) and revisit this after some while of collecting PRs and data (an iteration or two)?
There was a problem hiding this comment.
Update: the latest commit matches the phrasing I suggested above. Now, the only required usage details are to specify the tools used. But, there is still the option to describe a bit more, for anyone who wants to.
|
(There are two approving reviews but my suggestion would be to wait to merge this for a little while, maybe 1 week(?), to give everyone linked here a chance to take a look and hash things out in the discussion above.) |
erogluorhan
left a comment
erogluorhan
left a comment
There was a problem hiding this comment.
In addition to the inline comment below, I want to ask whether we are not creating a policy doc for this first iteration (either way is fine with me).
| <!-- If you did not use AI, please write "AI Usage: N/A" and remove these checklist items or mark as [N/A]. | ||
| Otherwise, please briefly specify all tools used and how they were used. Include model if known. Examples: | ||
| "AI Usage: Claude (Fable 5) made all code edits, but I came up with the ideas and design for this feature myself." | ||
| "AI Usage: discussion with Gemini and ChatGPT (5.5 Instant) to help find and understand the cause of this bug." |
There was a problem hiding this comment.
Per concerns around it would be discouraging, how about going lighter-weight with this first implementation (i.e. not ask detailed usage questions) and revisit this after some while of collecting PRs and data (an iteration or two)?
Disclosing tools used is still required, but describing how you used them is optional. See discussion in #1634 for more details.
No policy doc yet, let's address that in a different PR to keep this one small and easier to merge! Relevant text from original post in this PR:
|
|
Updating suggested plan now that there seems to be more agreement (to me it looks like the concerns have been addressed, and there are 2 approving reviews after all):
|
5 participants
Fixes part of #1630 but does not fully close it.
Overview
Adds a simple AI Disclosure section to PR form. Aiming for a relatively lightweight section here, in accordance with original issue description (#1630) and discussion (#1613).
This PR does not fully close the issue because it does not attempt to create an AI Usage Policy; that should be scoped to a different PR.
Also, unrelated to original issue, made a tiny rephrase for comment in "Expected Usage" section: now it says to delete the section (if it doesn't apply) instead of commenting it out. Reason for this change: if you're not super familiar with markdown syntax, it's easier to just delete something, rather than figuring out how to comment it out, and both outcomes are functionally the same here.
PR Checklist
General