FWIW, here's a successful workflow run with these scripts: https://github.com/treadmill-tb/tock/actions/runs/11350804975/job/31570037800

In case you can't view the logs, I posted the test-script ones to this gist: https://gist.github.com/lschuermann/174977a08a89adc419acf667b77bacb1

I would like to see this in a separate repository, for a couple reasons. First, the python code framework doesn't need to be reviewed like kernel code. Second, these tests need more than just the kernel to run. Third, these tests really shouldn't be changing with other kernel patches.

Other than that it's look good to me.

I agree with @bradjc. @charles37 we can discuss but can we close this PR and instead create a new repo for this in the tock organization?

I agree with @bradjc. @charles37 we can discuss but can we close this PR and instead create a new repo for this in the tock organization?

Yes, we can close this PR and create a separate repository for this. Thanks for the feedback!

@charles37 I created https://github.com/tock/tock-hardware-ci for this purpose, alongside @tock/hardware-ci-team, and given that team "maintain"-permissions on that repository. You should have received an invite to that team.

I like the idea of moving this to a different repository. We can use GitHub's reusable workflows to define a common, shared workflow in that repository and then import it in every repository that we want to run integration tests on. There's some guidance on how to import workflows from another repository here:
https://docs.github.com/en/actions/sharing-automations/reusing-workflows

We need to make sure that the workflow is parameterizable over which components should be fetched from their latest HEAD (aka., for kernel tests, that should be libtock-c, and which components we should take a local checkout from for the exact revision to test). The current code does not do this.

Do these tests block PRs from being merged?

Is there a way we can have some HWCI tests required and others not? I can imagine we would be able to detect which syscalls the userspace test uses, and maintain a list of stabilized syscall numbers, and if the test is subset then the test is required to pass. But for other tests we will be playing chicken and egg if the kernel PR changes the syscall interface for a non-stabilized syscall.

Do these tests block PRs from being merged?

Currently we do not. However, we arguably should.

Is there a way we can have some HWCI tests required and others not?

Yes. We can have this infrastructure work as follows:

1. A test-prepare phase will analyze changes and come up with a test strategy, in more or less the following form:

   boards:
     nrf52840dk_1:
       tests:
         multi_alarm_test:
           optional: false # fail the CI workflow if this test fails
        ble_passive_scanning:
          optional: true # simply issue a warning if this test fails

   This part is yet to be built. Currently, we generate a static such mapping, for a single nRF52840DK board.

2. We generate one GitHub actions test-execute job per such board (for BOARD in $BOARDS), which will execute the list of tests assigned to that board.

   We cannot dynamically create "steps" (foldable sections) in the job's output, or tolerate failures of individual tests.
   Thus we can create two steps:

   • Required tests, for which the job fails on any non-zero return code: for TEST in $REQ_TESTS; do $TEST; done
   • Optional tests, where failures are ignored: for TEST in $REQ_TESTS; do $TEST || echo "Oh no, to bad!"; done

However, I question the utility of optional tests if they just issue a warning, except for the case you mentioned (where, temporarily, kernel & userspace go out of sync). I don't think we'd actively look for warnings in CI runs.

Instead, I think that we should always mark all tests as required by default, and then manually make a PR to https://github.com/tock/tock-hardware-ci to temporarily mark a test as optional, and reenable it when both sides are in sync again. Would that alleviate your concerns? @bradjc

However, I question the utility of optional tests if they just issue a warning, except for the case you mentioned (where, temporarily, kernel & userspace go out of sync). I don't think we'd actively look for warnings in CI runs.

Yeah that's a good point. For most PRs we want all checks, and it should be well understood which PRs we expect to break checks.

Instead, I think that we should always mark all tests as required by default, and then manually make a PR to https://github.com/tock/tock-hardware-ci to temporarily mark a test as optional, and reenable it when both sides are in sync again. Would that alleviate your concerns? @bradjc

This seems complicated, but this is a complicated issue. I don't think we want to block progress on thinking about this at this point. It does seem like we are going to want to version all SyscallDrivers so the test suite knows when things are expected to be broken, or something like that, eventually. But for now we can just work around this challenge.

I've implemented the requested changes, and it is now ready for merge. The reusable workflow logic has been tested in tock/tock-hardware-ci, however the only way to test this workflow for the tock/tock specific environement is to have this PR enter the merge queue so it goes through the repository specific environment.

How does this compare to an upstream action like what we use for the labeler? https://github.com/tock/tock/blob/master/.github/workflows/labeler.yml

If this is effectively the same and if some day we publish the action and its an easy change to use it then this is great.

If it's different, then why don't we leave the full workflow in this repo? The tool (and python code) would be separate, still.

If it's different, then why don't we leave the full workflow in this repo? The tool (and python code) would be separate, still.

I don't understand the first part of the question, but the workflow and the scripts are very much tied together. For instance, the workflow will want to run some analysis steps based on the Python files, then schedule jobs, and then invoke these scripts.

By having this split out into a reusable workflow and putting it next to the Python code, we make sure that those things are always in sync. Note that this workflow "shim" in this repository can still parameterize the invocation of the underlying workflow.

Also, this deduplicates code we'd need to maintain in all the other repositories that we'd add this workflow to, such as libtock-c and libtock-rs.

It's different from "importing" an action: it won't run as a single step, but really insert this workflow into the parent one, and run it in the context as if it was located in this repository. This mechanism is only there for code deduplication. Here's how this will look like:

In a hypothetical future will we be able to have a workflow in Tock similar to this?

jobs:
  triage:
    permissions:
      contents: read  # for actions/labeler to determine modified files
      pull-requests: write  # for actions/labeler to add labels to PRs
    runs-on: ubuntu-latest
    steps:
    - uses: actions/[email protected]
      with:
        repo-token: "${{ secrets.GITHUB_TOKEN }}"

The key being uses: actions/[email protected]. If directly including a workflow file is a good analog to that without the commitment of upstreaming our github action then I support this. If directly including a workflow file is only a workaround for my earlier comment and not something we could ever do differently then I'm not so sure.

however the only way to test this workflow for the tock/tock specific environement is to have this PR enter the merge queue so it goes through the repository specific environment

I'll note that this restriction does not have anything to do with our use of reusable workflows, but just because we can't yet run Treadmill jobs on PRs from untrusted remote repositories or untrusted branches without potentially leaking secrets. This will be addressed in the future.

If directly including a workflow file is a good analog to that without the commitment of upstreaming our github action then I support this.

Yes, I think that including a reusable workflow is, modulo some UX differences, essentially analog to importing a published action, without needing to actually publish anything. The benefits of this being that we don't need to modify Tock (or any other repositories) for changes internal to the testing infrastructure.

In a hypothetical future will we be able to have a workflow in Tock similar to this?

Isn't this effectively what this PR is proposing right now?

We will probably not get rid of the base workflow in the tock-hardware-ci repository, because that is Tock-specific (but not specific to the kernel repository). We will also not want to use a published action, as that would mean that all the test output is collapsed into one "step" in that workflow execution.

In an ideal future, the structure we'd be aiming for is:

• tock/tock: imports tock/tock-hardware-ci (with some parameters)
• tock/libtock-c: imports tock/tock-hardware-ci (with some parameters)
• tock/tock-hardware-ci: uses some published Treadmill action, and includes Tock-specific test setup & harnesses