I tried using the workflow_job event instead of check_run. The webhook is ignoring the job though because it thinks that the labels do not match the configured labels for my runner.

2021-08-24T14:42:12.675Z	0b278f9f-b863-4236-b0f8-7a88a5b15e6a	DEBUG	Received workflow job event with labels: '["my-label"]'. The event does NOT match the configured labels: '
{}
'

The configured labels is empty, and I think it's an issue with the way RUNNER_LABELS is stored. runner_extra_labels is a comma separated list of labels. That variable is mapped to the RUNNER_LABELS environment variable for the lambda, but the workflow_job handler is expecting the RUNNER_LABELS environment variable to be a proper json array.

function isRunnerNotAllowed(job: WorkflowJob): boolean {
  const runnerLabelsEnv = (process.env.RUNNER_LABELS as string) || '[]';
  const runnerLabels = new Set(JSON.parse(runnerLabelsEnv) as Array<string>);