We probably want to remove dependabot from this, and maybe CodeQL as well. Not sure what harden-runner does, but I don't have a problem with specifying the hashes if it is best practice.

I generated this security PR to see what it did, motivated by the alerts in the security tab.

We probably want to remove dependabot from this, and maybe CodeQL as well. Not sure what harden-runner does, but I don't have a problem with specifying the hashes if it is best practice.

Hi @charris, thanks for using https://github.com/step-security/secure-workflows to create the PR. I am the maintainer for that project.

I am curious why you don't want to add CodeQL and Dependabot. Those changes increase the OpenSSF Scorecard score. Moreover, if dependabot.yml or some other updating mechanism is not used, the pinned hashes for Actions will not get updated (using a dependabot PR) when there is a new version.

w.r.t harden-runner, you can get more details here: https://github.com/step-security/harden-runner

To not add any of the best practices, you can uncheck the issue type before creating the PR at http://app.stepsecurity.io/securerepo. There is also a feature to preview the PR. Thanks!

Also adding @pnacht since I saw they initiated a discussion about Scorecard.

@varunsh-coder

We dropped dependabot because is was spamming all the numpy forks. It was a known problem, but it never got fixed AFAICT.

Not sure about CodeQL, it could be useful. If we dismiss an alert, does it stay dismissed?

We dropped dependabot because is was spamming all the numpy forks. It was a known problem, but it never got fixed AFAICT.

I see dependabot was removed in #20268, which was in November 2021. However, dependabot has recently become default-off for forks (November 2022).

@varunsh-coder

We dropped dependabot because is was spamming all the numpy forks. It was a known problem, but it never got fixed AFAICT.

Not sure about CodeQL, it could be useful. If we dismiss an alert, does it stay dismissed?

w.r.t CodeQL, yes, my experience has been that once an alert is dismissed in the Code Scanning UI, it stays dismissed.
https://docs.github.com/en/enterprise-cloud@latest/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/managing-code-scanning-alerts-for-your-repository#dismissing--alerts

I'm not sure I understand. Where did step-security-bot come from, who is behind it?

Can we make each addition a separate PR? I would like to think about each tool separately. Here are a few thoughts but again I think they should be dealt with one-at-a-time:

• dependabot: what are our runtime dependencies? We have build dependencies and test dependencies and documentation dependencies. I get that keeping runtime dependencies updated is good for security, but in what scenario is a DDOS vulnerability in a tool used in building docs a liability? When we had dependabot turned on, it generated weekly PRs that were really not helpful: they required time to research what changed and how it might break other parts of the project. Has the UX experience improved?
• Pinning dependencies in workflows to a hash instead of a version number: This feels weird. Why version software anymore, if we are supposed to link to hashes? Perhaps github actions should have clearer guidance here. When is a version number good enough and when is a hash required? How will we know when to update the hash?
• CodeQL: we have tried static code verification and we get various false positives along side helpful true positives. The suggestions require a deep dive in order to review and are quite time consuming. I don't know anything about CodeQL, but past attempts have felt more like we are beta testers for not-production-ready software than consumers of a valuable product. We run valgrind before a release. Did CodeQL actually find any actionable problems in the codebase? Is there a reason to prefer CodeQL over any of the other alternatives?

At the end of the day, the OpenSSF Scorecard is just another measure by a third-party service who may value certain measures of security without considering the maintainer burden they add. Does OpenSSF do any kind of cost-benefit analysis on their reccomendations? (I asked this question in ossf/scorecard#2627 Each new tool comes with more maintainer burden, more discussions about the validity of its recommendations. As more of these automated tools become available via bots, and more organizations develop measures that recommend their use, the maintainer workload keeps growing.

I'm not sure I understand. Where did step-security-bot come from, who is behind it?

Hi @mattip, this semi-automated PR was created using https://github.com/step-security/secure-workflows. I am the maintainer of that project. The project helps to apply security best practices using automation. Even though step-security-bot account created the PR, it can only be initiated by someone who has already contributed to a project. In this case, @charris used it to apply best practices using automation, whereas making these changes manually would have taken much longer. While creating PR, one can choose what best practices to apply, so it is possible to create separate PR for separate issue types. Hope this answers your questions. Please let me know if you have any other questions related to this.

Can we make each addition a separate PR? I would like to think about each tool separately. Here are a few thoughts but again I think they should be dealt with one-at-a-time:

• dependabot: what are our runtime dependencies? We have build dependencies and test dependencies and documentation dependencies. I get that keeping runtime dependencies updated is good for security, but in what scenario is a DDOS vulnerability in a tool used in building docs a liability? When we had dependabot turned on, it generated weekly PRs that were really not helpful: they required time to research what changed and how it might break other parts of the project. Has the UX experience improved?
• Pinning dependencies in workflows to a hash instead of a version number: This feels weird. Why version software anymore, if we are supposed to link to hashes? Perhaps github actions should have clearer guidance here. When is a version number good enough and when is a hash required? How will we know when to update the hash?

GitHub does recommend pinning 3rd party Actions. Dependabot can then update the hash when a new version is released. The tag of the new version is stored as a comment and that also gets updated by Dependabot.

• CodeQL: we have tried static code verification and we get various false positives along side helpful true positives. The suggestions require a deep dive in order to review and are quite time consuming. I don't know anything about CodeQL, but past attempts have felt more like we are beta testers for not-production-ready software than consumers of a valuable product. We run valgrind before a release. Did CodeQL actually find any actionable problems in the codebase? Is there a reason to prefer CodeQL over any of the other alternatives?

CodeQL is from GitHub https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql.

At the end of the day, the OpenSSF Scorecard is just another measure by a third-party service who may value certain measures of security without considering the maintainer burden they add. Does OpenSSF do any kind of cost-benefit analysis on their reccomendations? (I asked this question in ossf/scorecard#2627 Each new tool comes with more maintainer burden, more discussions about the validity of its recommendations. As more of these automated tools become available via bots, and more organizations develop measures that recommend their use, the maintainer workload keeps growing.

@pnacht feel free to add to what I wrote.

The project helps to apply security best practices using automation.

This concept is problematic. Security best practices cannot be applied without context. What in one area may be considered best practices may be overkill in other areas. Here is an example. A CVE was filed against setuptools claiming it could suffer from a DOS in a crafted package or custom PackageIndex page. But for the vast majority of the uses of setuptools (including the one in this repo NumPy), that CVE is not relevant since they do not use a custom PackageIndex. That CVE puts users of NumPy in an impossible situation: if they use automated tools to detect vulnerabilities, those tools will flag NumPy as unsafe and prevent its use. There currently is no released version of NumPy that can be used under that CVE. Well, except that:

• setuptools is only used for the build process of NumPy, not in the runtime.
• the use does not fall under the terms of the CVE since only PyPI is used as a PackageIndex

Ahh, but wait, what if someone sets an environment variable to target a private PackageIndex. Then is NumPy susceptible to the CVE or not? Well, then is the problem only going to affect NumPy or is it going to apply to the entire python environment they are using?

In short, in my opinion, adoption of automated tools to blindly apply security vulnerability data bases or automated tools for static code analysis to python code could have unintended consequences. For one such doomsday outcome, take a look at this discussion thread to ban python use in a corporate environment.

Security best practices cannot be applied without context.

@mattip I agree with you. I want to clarify that best practices are only being applied with the intent of a contributor. As you can see from @charris's comment,

I generated this security PR to see what it did, motivated by the alerts in the security tab.

Using https://github.com/step-security/secure-workflows, a contributor can choose what best practices they want to apply using a PR; each best practice can be de-selected using a checkbox. The contributor must then click on a button to generate a PR and can even create a preview of the PR to see the changes.

I hope that clarifies that the bot is not creating the PR independently with a fixed set of best practices. A contributor can choose what they think will help based on the context of the project and apply them. I'm sorry if this caused confusion, and if you believe the PR text can be improved to clarify this, I can take that action item. Please let me know. Thanks!

I see dependabot was removed in #20268, which was in November 2021. However, dependabot has recently become default-off for forks (November 2022).

That's great - finally, took them 3+ years - but please don't bother adding Dependabot back. They've been godawful, and until there is much more evidence that they have changed, I am a very hard -1 on re-enabling it.

each best practice can be de-selected using a checkbox.

I'll note that I de-selected some of the checks, saving them for later.

I am a very hard -1 on re-enabling it [dependabot].

Dependabot does run, you can see the results in the security tab, so it isn't like it is completely disabled. I used the results for #23130.

EDIT: Maybe we can set it up to only notify for security alerts. But we pretty much already have that with the security tab.

I am a very hard -1 on re-enabling it [dependabot].

Dependabot does run, you can see the results in the security tab, so it isn't like it is completely disabled.

It's like one of those prickly weeds in your garden - no matter how hard you work to get rid of it, you never fully succeed ...

you never fully succeed

I thought the security upgrades/pins were useful, it didn't bother alerting releases otherwise, which was nice. However, it did keep adding the same alerts to the list, so it has grown rather lengthy ...

If the code quality comments mainly/only show up as annotations on changed lines, I am happy with them, should be rare enough to ignore them (i.e. none of the ones currently there seem really important, a more efficient regex may be nice, but really isn't a priority).
I do agree that historically issues being opened via the use of static tools were likely more churn than useful.

No opinion on dependabot, pinning hashes seems fine in general (seems a bit like it makes more sense with rare auto-updates). Would it be possible to schedule dependabot on a very slow schedule to keep the hashes up-to date but otherwise hear nothing from it for months at a time?

Would it be possible to schedule dependabot on a very slow schedule

I'm happy to leave it under the security tab, but we do need to take a look now and then.

should be rare enough to ignore them

You can dismiss them and they should stay dismissed. That said, I think we could make issues of the current ones, they could be improved for clarity if nothing else.

Harden-Runner is a bit of a mystery to me and is a third party app. More information is at https://github.com/marketplace/actions/harden-runner.

Harden-Runner is a bit of a mystery to me and is a third party app. More information is at https://github.com/marketplace/actions/harden-runner.

@charris I am also the maintainer for harden-runner, so let me know if you have any questions about it.
It gives a view of outbound traffic and source code overwrite during the job run, and then suggests and enables setting a policy to restrict outbound traffic and run job without sudo access.
As an example, here is the insights view for one of the jobs in this PR:
https://app.stepsecurity.io/github/numpy/numpy/actions/runs/4047433631
These insights links are available in the job summary and build log, e.g.
https://github.com/numpy/numpy/actions/runs/4047433631/jobs/6961444166#step:3:12

Will regenerate to resolve conflicts.