@tylerjereddy is the codecov use in tools/travis-test.sh still useful, or is it a left-over? The data in https://codecov.io/gh/numpy/numpy/tree/master/numpy doesn't have any Python code, so I assume no one is looking at this or we'd have noticed.

@rgommers if that's true, I'd be happy to help to see how we can make it useful for you. Codecov is free and will always be free for open-source.

Thanks @thomasrockhu. The reason we aren't looking anymore is that your bot was so noisy and persistent that we had to block it from posting any comments. If we try the new app integration and that's nicer, we know how to fix uploading the Python coverage results.

@rgommers I totally understand. If the bot is being too noisy, you can always mute it by creating a codecov.yml file in the root with

comment: false

Let me know if that is what you're looking for, or if that's not what you had meant.

yeah we tried, that didn't stop it in the past :)

The GitHub app integration replaces the bot I hope?

Yes, the app integration replaces the bot. As for the yaml issue, I'm sorry to hear it didn't work in the past. I'm happy to help you fix that if it isn't working as expected.

I was noticing this lately, and that would be great to get codecov up and running again in a more accessible/discoverable way. Its running, but I guess we probably don't check it as much as would be useful juts because its a bit hidden...

Lets discuss it in the meeting briefly, but I really would like to set this up ASAP even if it requires some iteration to get the noise level to where we want it to be. There are a few larger PRs open, and having code coverage on the PRs would help especially on the C-side.

is the codecov use in tools/travis-test.sh still useful,

@rgommers I think it may still be useful--it is basically copied from the work @pv did to incorporate compiled language coverage into codecov reports in SciPy CI.

I suspect if the app "just works" (or, with a bit of iteration), then the same gcov results funneled to codecov will integrate with Python coverage the same way they did before.

I can't make the meeting, but +1 for reactivation of the service (in NumPy & SciPy) now that we have a responsive contact. We saw that having a responsive contact has also been quite productive for swiftly resolving Azure CI matters as well.

I installed the GitHub App, @seberg and/or @tylerjereddy please go ahead and play/tweak/fix:)

Standing at the ready for any questions and feedback!

It looks like everyone is satisfied with the current setup - if there are any problems in the future please feel free to tweak the configuration or re-open this for further discussion!

@thomasrockhu in case you can help us out and point to the problem right away (if there is one):
https://github.com/numpy/numpy/pull/19434/files#diff-6b1b23de139c1e509887d2dcd454bf03e431817eaef61e78497658fffb52067a

Marks a lot of lines as "Added line #L685 was not covered by tests" also for files that are not even touched (below). Getting some additional codecov annotations is sometimes nice, but it feels like it is getting too much. And I am not sure why it is happening, the "Unchanged files with check annotations" may often be fairly new files added to the repository, but even then, I think the PRs are based off the branch including them. Or is there some weird mixup in our setup because of the master -> main rename?

@seberg, nothing comes immediately to mind (but I took a superficial look) except for the possible branch rename. I'll dig into this a little later today if that's ok

It is confusing me. If I go on the codecov website and select the main branch and the commit in question, the result looks good. But the codecov integration always says: codecov/project — 83.02% (-0.30%) compared to 004ad40 and 004ad40 is a month old commit... (e.g. here: #19422) and if I click on the codecov patch link, I always get an error.

So that is probably be the issue, but I am not sure why it is using that specific commit every time? And I am not aware of anything changed since that commit happened.

The base option looks like it may be mildly useful: https://docs.codecov.com/docs/commit-status#base but it says it is deprecated.

The branch rename was a thought. But honestly, it happened much earlier than the commit codecov is comparing too (or the much too verbose output). So maybe it has something to do with things, but I don't think it is stuck on master.

OK, wrapping up my understanding:

• Codecov reports things correctly on commits against the maintanence/1.21.x branch, but not against main (maybe due to the rename?)
• We do upload the data for the main branch: https://codecov.io/gh/numpy/numpy/branches/main seems perfectly fine.
• For PRs against main, codecov reports for example: Changes found in between 004ad40...b32b72e (pseudo...base) where b32b72e is the correct base.

I am suspecting that the "pseudo" 004ad40 is just nonsense, that is a valid PR from a long time ago. So I think what is wrong is that codecov somehow uses the incorrect pseudo/merge commit for the PR.

I assume this has something to do with the incorrect merge commit being reported to codecov as the PRs merge commit hash. But I have no clue at all why?

This looks like a similar problem: https://community.codecov.com/t/nearest-pull-request-base-pseudobase-commit-not-chosen-correctly/2249/4

@thomasrockhu it would be awesome if someone at codecov could look into why the base pseudo commit for the PR comparison is always listed as 004ad40. If we figure out where that commit comes from, it feels like we should know what is going on.

Hi @seberg, apologies I wasn't able to come to any conclusions last week. I will continue to work out why this is happening on our side, and thanks for the patience.

@seberg, just a quick update, I'm not sure what is happening in these cases and why 004ad40 keeps popping up. I've pulled in the product team to help take a look.

@seberg, apologies, this is really stumping us. I've escalated again to try to figure out why this is happening.

Thanks! Let me know if there is anything we can try in NumPy.

@seberg, looks like this uncovered a pretty difficult bug. Thanks for helping us identify! We believe we have found the root cause and are working on a fix.

@seberg we have deployed the change. Let me know if you see this in any future PRs.

Thanks, hopefully it works, then! I guess we may also need to have a new commit on the main branch and the PR based off that? Anyway, I guess we will see, even if that is necessary, I guess things should normalize by Monday then.

Aha, gh-19547 is correct already, so that is good :). Although the Codcov overview still says "Missing base report": https://app.codecov.io/gh/numpy/numpy/pulls?page=1&state=open&order=-pullid but lets see and wait I guess.

I somewhat expect this was also the problem here: https://community.codecov.com/t/nearest-pull-request-base-pseudobase-commit-not-chosen-correctly/2249/4 just in case you want to make a note there.

Ok, please ping back here if it hasn't normalized by then and thanks for your patience!