Add agent: research harness engineer - #2854
Conversation
🔒 PR Risk Scan ResultsScanned 1 changed file(s).
✅ No matching risk patterns were detected in changed files.
|
aaronpowell
left a comment
There was a problem hiding this comment.
I wonder if instructions are the right way to be adding this, maybe a custom agent would be a better approach.
The risk the instructions provide is that they are applied broadly on any file that is being worked on. but it reads a bit like the goal is to steer the agent down a thinking path, which would be better from the agent definition
|
Good call — agreed that the content is steering a way of thinking rather than per-file conventions, and broad applyTo was the weak point. Converted to a custom agent: |
🔍 Vally Lint Results
Summary
Full linter output |
aaronpowell
left a comment
There was a problem hiding this comment.
can you re-run the script to update the readme, there seems to be an accidental change included
|
Done — docs/README.agents.md is now upstream plus only the new agent row; the accidental generator churn (live MCP-registry formatting) is gone. |
|
@all-contributors add @GuoCheng24 for agents |
|
I couldn't determine any contributions to add, did you specify any contributions? |
|
@all-contributors please add @GuoCheng24 for agents |
|
I couldn't determine any contributions to add, did you specify any contributions? |
|
@all-contributors add @GuoCheng24 for content |
Adds
instructions/research-experiment-discipline.instructions.md— guidance for repositories where Copilot writes evaluation harnesses, benchmark sweeps, or any code that reports a performance number.Falls under "Development methodologies and best practices" / "Testing strategies and quality assurance" in CONTRIBUTING. The gap it addresses: in research code the common failure is not a crash but a number that looks great and is wrong — models happily write evaluation code that tunes on the test set, compares against unmatched baselines, or reports point estimates inflated by group leakage. The instructions are specific and mechanical: null models scored first (freeze on a well-scoring null), calibration/held-out separation on the unit of independence, a held-out access budget, published-baseline reproduction before improvement claims, paired error bars, and every new check demonstrated to fail on a deliberately broken input.
Distilled from real failures in agent-run experiment campaigns; the longer-form methodology with runnable demonstrations lives at https://github.com/GuoCheng24/breakthrough-harness.
mainnpm startrun;docs/README.instructions.mdupdated (the script also produced unrelated churn indocs/README.agents.mdfrom live MCP-registry data — left out to keep the diff minimal; happy to include if the check requires it)