Lots of ci/prow lanes are failing due to gcloud problem #6405

jlojosnegros · 2022-11-29T14:42:00Z

jlojosnegros
Nov 29, 2022

Hi all,

I am working on a PR and have seen some ci/lanes failing due to a problem with gcloud.
I have been looking around and I have notice is also failing on many other PRs.

I would like to know if this is an already known issue and if so, if there is a known solution I can apply or if, on the other hand some investigation is needed.

Thanks in advance

Activated service account credentials for: [do-not-delete-ci-provisioner@XXXXXXXXXXXXXXXXXXXXXX.iam.gserviceaccount.com]
Updated property [core/project].
You do not currently have this command group installed.  Using it 
requires the installation of components: [alpha]
ERROR: (gcloud) You cannot perform this action because you do not have permission to modify the Google Cloud SDK installation directory [/usr/local/gcloud/google-cloud-sdk].
Re-run the command with sudo: sudo /usr/local/gcloud/google-cloud-sdk/bin/gcloud ...

jlojosnegros · 2022-11-29T14:52:11Z

jlojosnegros
Nov 29, 2022
Author

I have also notice that kata-containers ci lane is failing due to an error in the golang version.

Repo /tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/kata-containers already in git's safe.directory
/tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/kata-containers/src/runtime /tmp/jenkins/workspace/kata-containers-2-crio-PR/go/src/github.com/kata-containers/tests
golang.mk:60: *** "ERROR: golang minor version too old: got 1.18.1, need atleast 1.19".  Stop.

2 replies

littlejawa Nov 29, 2022

Yes. We are experiencing issues on the kata CI side, and this is a side effect. We are working on it, but other issues need to be fixed before we can make this pass.

littlejawa Nov 30, 2022

I have made a quick fix to unblock PRs, it should be testing correctly now.
A cleaner patch is on its way on the kata side, but that should be transparent for the cri-o devs.
Sorry for the delay.

haircommander · 2022-11-29T16:08:45Z

haircommander
Nov 29, 2022
Maintainer

interestingly, that's actually not the failure. for some reason, prow finds that abnormal and posts it, but the failure is farther down:

Container test exited with code 1, reason Error
---
ble.builtin.seboolean to ansible.posix.seboolean
Using module file /home/deadbeef/.local/lib/python3.9/site-packages/ansible_collections/ansible/posix/plugins/modules/seboolean.py
<localhost> PUT /home/deadbeef/.ansible/tmp/ansible-local-57899fa_6bdea/tmp6v3hsvwu TO /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/ /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-kmhhazrdresplefzcydljawiqjlkcwza ; GOPATH=/usr/go /usr/bin/python3 /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/AnsiballZ_seboolean.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/deadbeef/.ansible/tmp/ansible-tmp-1669733325.0824344-65783-279178949064296/ > /dev/null 2>&1 && sleep 0'

TASK [Containers using cgroups and/or systemd after 7.5 are permitted] *********
task path: /home/deadbeef/cri-o/contrib/test/ci/system.yml:65
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "ignore_selinux_state": false,
            "name": "container_manage_cgroup",
            "persistent": true,
            "state": true
        }
    }
}

MSG:

SELinux boolean container_manage_cgroup does not exist.

something is funky in the container-selinux package on rhel 9. @sohankunkerkar is looking into it

6 replies

haircommander Jan 24, 2023
Maintainer

I am pretty sure this particular error is fixed now, where are you seeing it?

jlojosnegros Jan 25, 2023
Author

Yeah sorry. It fails but with different errors, my bad.

Still here #6388

kata-containers
ci-fedora-integration
ci-rhel-integration

are failing recurrently

kata-containers seems to be failing due to a timeout in here...

Signature ok
subject=CN = Kata Containers Webhook
Getting CA Private Key
secret/pod-annotate-webhook-certs created
mutatingwebhookconfiguration.admissionregistration.k8s.io/pod-annotate-webhook created
deployment.apps/pod-annotate-webhook created
service/pod-annotate-webhook created
[webhook-check.sh:70] INFO: Going to check the kata-webhook installation
ERROR: The pod-annotate-webhook deployment is unavailable after 60s waiting

ci-fedora-integration and ci-rhel-integration are failing while trying to run localintegration

TASK [run integration tests] ***************************************************
task path: /home/deadbeef/cri-o/contrib/test/ci/test.yml:61
fatal: [localhost]: FAILED! => {
    "ansible_job_id": "764668902847.128257",
    "changed": true,
    "cmd": "make localintegration 2>&1 > /tmp/artifacts/testout.txt",
    "delta": "0:07:18.327718",
    "end": "2023-01-25 09:24:58.315529",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_raw_params": "make localintegration 2>&1 > /tmp/artifacts/testout.txt",
            "_uses_shell": true,
            "argv": null,
            "chdir": "/usr/go/src/github.com/cri-o/cri-o",
            "creates": null,
            "executable": "/bin/bash",
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "rc": 2,
    "results_file": "/tmp/artifacts/async/764668902847.128257",
    "start": "2023-01-25 09:17:39.987811",
    "started": 1
}

but I am not sure what the problem is but it seems unlikely to me that it can be related with the change in the PR.

I've been looking at the latests PR merged and it seems this ci-lanes have been overridden but I do not know if it was due to the same failure or not.

P.S.:
here are the latest run of each of the lanes

haircommander Jan 25, 2023
Maintainer

thanks for mentioning! we have issue open for them: #6523 #6522 #6481
All we're aware of and in various stages of fixing

jlojosnegros Jan 30, 2023
Author

Do you think #6388 could be merged overriding the ci-lanes or should we wait?

haircommander Jan 30, 2023
Maintainer

yeah definitely :)

Lots of ci/prow lanes are failing due to gcloud problem #6405

Uh oh!

jlojosnegros Nov 29, 2022

Replies: 2 comments · 8 replies

Uh oh!

jlojosnegros Nov 29, 2022 Author

Uh oh!

littlejawa Nov 29, 2022

Uh oh!

littlejawa Nov 30, 2022

Uh oh!

haircommander Nov 29, 2022 Maintainer

Uh oh!

haircommander Jan 24, 2023 Maintainer

Uh oh!

Uh oh!

jlojosnegros Jan 25, 2023 Author

Uh oh!

haircommander Jan 25, 2023 Maintainer

Uh oh!

jlojosnegros Jan 30, 2023 Author

Uh oh!

haircommander Jan 30, 2023 Maintainer

jlojosnegros
Nov 29, 2022

Replies: 2 comments 8 replies

jlojosnegros
Nov 29, 2022
Author

haircommander
Nov 29, 2022
Maintainer

haircommander Jan 24, 2023
Maintainer

jlojosnegros Jan 25, 2023
Author

haircommander Jan 25, 2023
Maintainer

jlojosnegros Jan 30, 2023
Author

haircommander Jan 30, 2023
Maintainer