-
-
Notifications
You must be signed in to change notification settings - Fork 989
Update guidelines on using GenAI #1778
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
726ec3c
e71105b
f8ab6ab
a04cd52
ef31ff8
ff94cd6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -9,6 +9,24 @@ contribution is the responsibility of the contributor. We value good code, conci | |||||||||
| and avoiding unneeded code churn. Discretion, good judgment, and critical thinking are the foundation of all good | ||||||||||
| contributions, regardless of the tools used in their creation. | ||||||||||
|
|
||||||||||
| Considerations for success | ||||||||||
| ========================== | ||||||||||
|
|
||||||||||
| Authors must review the work done by AI tooling in detail to ensure it actually makes sense before proposing it as a PR. | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| We expect PR authors to be able to explain their proposed changes in their own words. | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| Disclosure of the use of AI tools in the PR description is appreciated, while not required. Be prepared to explain how | ||||||||||
| the tool was used and what changes it made. | ||||||||||
|
Comment on lines
+19
to
+20
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Looks like some funky line breaking?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I had it to break after 120 characters. But now that I read the devguide's Rst markup doc, seems like we're supposed to break at 80 characters. |
||||||||||
|
|
||||||||||
| Whether you are using Generative AI or not, keep the following principles in mind for the quality | ||||||||||
| of your contribution: | ||||||||||
|
Comment on lines
+22
to
+23
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| - Consider whether the change is necessary | ||||||||||
| - Make minimal, focused changes | ||||||||||
| - Follow existing coding style and patterns | ||||||||||
| - Write tests that exercise the change | ||||||||||
|
Comment on lines
+25
to
+28
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we add another bullet point along the lines of: perhaps a follow paragraph after this list: "Pay close attention to your AI's testing behavior. Have conversations with your AI model about the appropriateness of changes given these principles before you propose them."
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would rather that we not personify the tools @gpshead. Perhaps: "Pay close attention to an AI tool's recommendations for testing changes. Provide input about Python's testing principles before requests to the AI tool's model. Always review the AI tool's output before opening a pull request or issue." |
||||||||||
|
|
||||||||||
| Acceptable uses | ||||||||||
| =============== | ||||||||||
|
|
||||||||||
|
|
@@ -28,21 +46,3 @@ If a contributor repeatedly opens unproductive issues or PRs, they may be blocke | |||||||||
|
|
||||||||||
| Sometimes AI assisted tools make failing unit tests pass by altering or bypassing the tests rather than addressing the | ||||||||||
| underlying problem in the code. Such changes do not represent a real fix and are not acceptable. | ||||||||||
|
Comment on lines
+47
to
+48
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd like to see this worded in more general terms rather than using such a specific example (older models did this a lot more than 2026's). What this is really getting at is that we want people to be cautious about reward hacking rather than addressing the actual underlying problem in a backwards compatible manner. maybe something along the lines of: "Some models have had a tendency of reward hacking by making incorrect changes to fix their limited context view of the problem at hand rather than focusing on what is correct. Including altering or bypassing existing tests. Such changes do not represent a real fix and are not acceptable."
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this can be generalized beyond AI tools to humans as well.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Some AI tools may provide responses to a user's prompt that diverge from recommended practices since the AI tool may not have been trained on the full context of the problem and recommended practices. Sometimes, due to limited context, the tool will alter or bypass existing tests. Such changes do not offer a real fix and are not acceptable."
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd avoid using the word "trained" as that has a specific meaning in the AI field that isn't really the reason. Focusing on "context" is good as that's the important and widely known term used in AI. Just "... the AI tool may not have the full context of the problem and recommended practices. You need to provide it that.". I've never really liked the "Sometimes, due to limited context, the tool will alter or bypass existing tests." example as it is dated for anyone using the latest models (not everyone is... an entirely different access problem that thus makes general purpose vague docs like this hard). But it felt like we should keep some form of an example undesirable behavior from an insufficiently guided model in here in order to make the more important "Such changes do not offer a real fix and are not acceptable." be tied to a concrete example. So absent clearly better ideas, and knowing some users will be using lesser models, it still fits.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Agreed. Proposing: Due to limited context, an AI tool may alter or bypass existing tests. Such changes do not offer a real fix and are not acceptable."
Comment on lines
+47
to
+48
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
|
||||||||||
| Considerations for success | ||||||||||
| ========================== | ||||||||||
|
|
||||||||||
| Authors must review the work done by AI tooling in detail to ensure it actually makes sense before proposing it as a PR. | ||||||||||
|
|
||||||||||
| We expect PR authors to be able to explain their proposed changes in their own words. | ||||||||||
|
|
||||||||||
| Disclosure of the use of AI tools in the PR description is appreciated, while not required. Be prepared to explain how | ||||||||||
| the tool was used and what changes it made. | ||||||||||
|
|
||||||||||
| Whether you are using Generative AI or not, keep the following principles in mind for the quality | ||||||||||
| of your contribution: | ||||||||||
|
|
||||||||||
| - Consider whether the change is necessary | ||||||||||
| - Make minimal, focused changes | ||||||||||
| - Follow existing coding style and patterns | ||||||||||
| - Write tests that exercise the change | ||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.