This looks like something an LLM would spit out when you ask for a review - is it necessary to include this in the instructions?

Ah, this whole section looks like it is something an LLM spit out as an execution plan. Should something like this be hand-rolled with explicit instructions on how to conduct the review and important considerations?

I guess my curiosity here is if we made an explicit guidelines type document that humans could follow, an LLM should be able to follow along relatively easily and we can potentially enforce more nuanced guardrails.

Not worth blocking this on - more so asking questions for myself as I've not worked with LLMs in this capacity before.

This is an artefact of how these documents are built. Using Claude locally, I've given it text based prompts, and it has converted that into instructions that it can read. So 95% of this document is it translating my instructions and feedback into rules that it can later apply.

Hmm. That is interesting. So this file is automatically updated with more detailed instructions by Claude as the AGENTS.md file is updated?

If you update this file by hand to "improve it", what happens?

You can improve it by hand, that's also fine and I have made a few edits here and there. But the bulk was generated, (then pruned - it was more verbose), and tested again to see if it was producing good results.

is it necessary to include this in the instructions?

I've found giving explicit examples, when you want a particular format of output to work quite well. Unfortunately, it seems best with monkey see monkey do style learning in this department :(

I guess my curiosity here is if we made an explicit guidelines type document that humans could follow, an LLM should be able to follow along relatively easily and we can potentially enforce more nuanced guardrails.

I think this should work, and is sort of what the section ### API review in AGENTS.md is, but you would still need something similar to this document for the command to give it a guide on how you want the output, and communication style alongside extra prompts for when it fails to follow the rules...

If you update this file by hand to "improve it", what happens?

I'be found that often, what makes sense to me (as a human, reading the document), the AI will ignore / may degrade performance. Conversing with the model and asking it what seems to be the issue (why didn't you listen to my instructions?) does seem to help - but yeilds different structures to what we'd normally expect e.g for documentation to be consumed by humans.

This does lead into one of the bigger problems with automating problems like this, which is how do you ensure consistent output, and avoid regressions when experimenting with prompts? You've got a probabilistic system you need to build confidence in, and don't want to rely on manual checks or 'vibes'. evals are the way the industry seems to be moving, but the OSS tooling all seems pretty heavyweight for automating internal tasks / reviews.

One approach for API review that may work could be directing the command (or a version of it) to output json:

{
  "summary": "…",
  "issues": [
    {"rule": "fieldDocumentation",  "msg": "...", "lines": "…"},
    {"rule": "optionalFieldBehaviour", "msg": "...", "lines": "…"}
  ],
  "meta": {"model": "claude-xxx",  "rules_version": "<commit?>"}
}

Having sets of real API PRs, where we can write units that Expect the correct issues to be caught. Something like:

golden/: real API doc chunks + your expected issue set (ground truth).
synthetic/: synthetic snippets each targeting exactly one rule. These are your 'unit tests' .

Then you can compare existing rules/prompt to changed, and hopefully catch any changes? There are lots of ways to extend this too e.g mutating synthetic snippets, and still expecting issues to be caught, or using existing reviews (e.g merged PRs with comment chains + changes requested) and an llm as a judge style system.

Either way, its not simple... unfortunately :(

Do we need to create a separate H1 heading to explain to the LLM that this is the steps for how to actually conduct the review?

Weirdly, I'll run a comprehensive API review for the OpenShift API changes in the specified GitHub PR. seems to work just fine. I've had instances (e.g on ccapio) where adding headings lead to the agent ignoring them 🤦

It's the sort of thing you want to build something to allow you to test :/

Should we teach it how to run the integration tests with more focused arguments?

That way running the integration tests don't take longer than necessary when reviewing a change that only impacts a subset of the APIs/tests?

Yes, that's a good idea, I'll have a go at that