@andrewleech I just saw you tackled this exact problem in #12658, but I think there are still some scenarios where it still checks the wrong commits.

Thanks, this looks good to me, and your testing is thorough!

As you noted, do we also need to update the related logic in ci_code_size_build?

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.57%. Comparing base (d34b15a) to head (9591b0a).
Report is 2 commits behind head on master.

@@           Coverage Diff           @@
##           master   #16114   +/-   ##
=======================================
  Coverage   98.57%   98.57%           
=======================================
  Files         164      164           
  Lines       21345    21345           
=======================================
  Hits        21040    21040           
  Misses        305      305           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

That looks great @projectgus , thanks for the clear description both here and in the script.
I'm pretty sure I never realised GitHub ran its actions in a trial merge, that does complicate it just a little bit more!

Oh the existing comment did mention the merge and try to exclude it, regardless I like the look of your changes and testing.

I'm pretty sure I never realised GitHub ran its actions in a trial merge, that does complicate it just a little bit more!

Yeah, I hadn't realised either until I looked at this. There is a way to tell the GitHub checkout action to check out the PR branch HEAD instead of the merge commit, but this actually gives us more work in this case because we still don't know what the merge base is, and we need this to know which commits to check! Whereas having the dummy merge commit (in this case) means we have all of the git history we need without needing to fetch anything new.

As you noted, do we also need to update the related logic in ci_code_size_build?

I looked at this yesterday and thought it was OK, but I just looked again and I think there's a bug: it currently compares the PR branch's merge base against the generated merge commit. So that means any changes merged on master which aren't in the PR branch are included in the size comparison.

I think it should either compare the merge base with the PR branch HEAD, or compare the generated merge commit with the master branch. Probably the second one as that's exactly what the code size increase would be if the PR was merged.

Will submit another PR for this.