Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@kolyshkin
Copy link
Collaborator

@kolyshkin kolyshkin commented Feb 25, 2021

Debugging the failure of integration_crun job in #4564.

It currently fails:

TASK [run integration tests] ***************************************************
task path: /go/src/github.com/cri-o/cri-o/contrib/test/integration/test.yml:21
fatal: [localhost]: FAILED! => {"ansible_job_id": "72200651440.24472", "changed": true, "cmd": "make localintegration >& /tmp/artifacts/testout.txt", "delta": "0:07:13.553467", "end": "2021-02-25 23:22:51.995499", "finished": 1, "msg": "non-zero return code", "rc": 2, "start": "2021-02-25 23:15:38.442032", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

but the details of the failure as in testout.txt, which is apparently being uploaded:

{"component":"gcsupload","file":"prow/gcsupload/run.go:192","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts.func1","level":"info","msg":"Found /data/gcs/artifacts/testout.txt in artifact directory. Uploading as pr-logs/pull/cri-o_cri-o/4564/test_pull_request_crio_integration_crun_fedora/14001/artifacts/testout.txt\n","severity":"info","time":"2021-02-25T23:23:40Z"}
{"component":"gcsupload","dest":"pr-logs/pull/cri-o_cri-o/4564/test_pull_request_crio_integration_crun_fedora/14001/artifacts/journals/systemd-journald.service","file":"prow/pod-utils/gcs/upload.go:88","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2021-02-25T23:23:40Z"}
{"component":"gcsupload","dest":"pr-logs/pull/cri-o_cri-o/4564/test_pull_request_crio_integration_crun_fedora/14001/artifacts/testout.txt","file":"prow/pod-utils/gcs/upload.go:88","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2021-02-25T23:23:40Z"}

but I can't see it in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-federated-results/pr-logs/pull/cri-o_cri-o/4564/test_pull_request_crio_integration_crun_fedora/14001/

thus this hack to show the results directly to stdout.

@openshift-ci-robot
Copy link

@kolyshkin: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Feb 25, 2021
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kolyshkin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Feb 25, 2021
@openshift-ci-robot

This comment has been minimized.

@kolyshkin kolyshkin changed the base branch from master to release-1.19 February 25, 2021 23:49
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 25, 2021

@kolyshkin: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/e2e-agnostic 1a5b907 link /test e2e-agnostic
ci/prow/e2e-gcp 1a5b907 link /test e2e-gcp

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kolyshkin
Copy link
Collaborator Author

This is funny, now that I see the test results it actually succeds:

[kir@kir-rhat cri-o]$ sed 's/", "/\n/g' < eee
changed: [localhost] => {"ansible_job_id": "77878671629.24465
changed": true, "cmd": "make localintegration 2>&1 | tee /tmp/artifacts/testout.txt
delta": "0:06:52.355308
end": "2021-02-26 00:04:05.368154
finished": 1, "rc": 0, "start": "2021-02-25 23:57:13.012846
....
make[1]: Leaving directory '/go/src/github.com/cri-o/cri-o/pinns'
GO111MODULE=on go build --mod=vendor  -ldflags '-s -w -X github.com/cri-o/cri-o/internal/pkg/criocli.DefaultsPath=\"\" -X github.com/cri-o/cri-o/internal/version.buildDate='2021-02-25T23:57:24Z' -X github.com/cri-o/cri-o/internal/version.gitCommit=1a5b907e5b40e42586774606ad400de225482fce -X github.com/cri-o/cri-o/internal/version.gitTreeState=dirty ' -tags \"containers_image_ostree_stub  exclude_graphdriver_btrfs btrfs_noversion    seccomp selinux\" -o test/copyimg/copyimg github.com/cri-o/cri-o/test/copyimg
GO111MODULE=on go build --mod=vendor  -ldflags '-s -w -X github.com/cri-o/cri-o/internal/pkg/criocli.DefaultsPath=\"\" -X github.com/cri-o/cri-o/internal/version.buildDate='2021-02-25T23:57:26Z' -X github.com/cri-o/cri-o/internal/version.gitCommit=1a5b907e5b40e42586774606ad400de225482fce -X github.com/cri-o/cri-o/internal/version.gitTreeState=dirty ' -tags \"containers_image_ostree_stub  exclude_graphdriver_btrfs btrfs_noversion    seccomp selinux\" -o test/checkseccomp/checkseccomp github.com/cri-o/cri-o/test/checkseccomp
./test/test_runner.sh 
++ time bats --jobs 16 --tap .
1..175
ok 1 apparmor tests (in sequence) # skip skip test since apparmor is not enabled.
ok 2 pids limit
ok 3 conmon custom cgroup
ok 4 crio commands
ok 5 invalid ulimits
ok 6 invalid devices
ok 7 invalid metrics port
ok 8 invalid log max
ok 9 log max boundary testing
ok 10 config dir should succeed
ok 11 config dir should fail with invalid option
ok 12 config migrate should succeed with default config
ok 13 config migrate should succeed with 1.17 config
ok 14 config migrate should fail on invalid version
ok 15 remove containers and images when remove both
ok 16 remove containers when remove temporary
ok 17 clear neither when remove persist
ok 18 don't clear podman containers # skip Podman not installed
ok 19 run the critest suite # skip critest because RUN_CRITEST is not set
ok 20 ctr not found correct error message
ok 21 ctr termination reason Completed
ok 22 ctr termination reason Error
ok 23 ulimits
ok 24 additional devices support
ok 25 additional devices permissions
ok 26 ctr remove
ok 27 ctr lifecycle
ok 28 ctr logging
ok 29 ctr journald logging
ok 30 ctr logging [tty=true]
ok 31 ctr log max
ok 32 ctr log max with default value
ok 33 ctr log max with minimum value
ok 34 ctr partial line logging
ok 35 ctrs status for a pod
ok 36 ctr list filtering
ok 37 ctr list label filtering
ok 38 ctr metadata in list & status
ok 39 ctr execsync conflicting with conmon flags parsing
ok 40 ctr execsync
ok 41 ctr execsync should not overwrite initial spec args
ok 42 ctr device add
ok 43 privileged ctr device add
ok 44 privileged ctr add duplicate device as host
ok 45 ctr hostname env
ok 46 ctr execsync failure
ok 47 ctr execsync exit code
ok 48 ctr execsync std{out,err}
ok 49 ctr stop idempotent
ok 50 ctr caps drop
ok 51 ctr with default list of capabilities from crio.conf
ok 52 ctr with list of capabilities given by user in crio.conf
ok 53 run ctr with image with Config.Volumes
ok 54 ctr oom
ok 55 ctr /etc/resolv.conf rw/ro mode
ok 56 ctr create with non-existent command
ok 57 ctr create with non-existent command [tty]
ok 58 ctr update resources
ok 59 ctr correctly setup working directory
ok 60 ctr execsync conflicting with conmon env
ok 61 ctr resources
ok 62 ctr with non-root user has no effective capabilities
ok 63 ctr with low memory configured should not be created
ok 64 ctr expose metrics with default port
ok 65 ctr expose metrics with custom port
ok 66 privileged ctr -- check for rw mounts
ok 67 annotations passed through
ok 68 ctr with default_env set in configuration
ok 69 ctr seccomp profiles unconfined
ok 70 ctr seccomp profiles runtime/default
ok 71 ctr seccomp profiles unconfined by empty field
ok 72 ctr seccomp profiles wrong profile name
ok 73 ctr seccomp profiles localhost/profile_name
ok 74 ctr seccomp profiles docker/default
ok 75 ctr_userns run container
ok 76 bind secrets mounts to container
ok 77 default mounts correctly sorted with other mounts
ok 78 test deprecated --default-mounts flag
ok 79 test infra ctr dropped
ok 80 test infra ctr not dropped
ok 81 pod test hooks
ok 82 run container in pod with image ID
ok 83 container status when created by image ID
ok 84 container status when created by image tagged reference
ok 85 container status when created by image canonical reference
ok 86 container status when created by image list canonical reference
ok 87 image pull and list
ok 88 image pull with signature # skip registry has some issues
ok 89 image pull and list by tag and ID
ok 90 image pull and list by digest and ID
ok 91 image pull and list by manifest list digest
ok 92 image pull and list by manifest list tag
ok 93 image pull and list by manifest list and individual digest
ok 94 image pull and list by individual and manifest list digest
ok 95 image list with filter
ok 96 image list/remove
ok 97 image status/remove
ok 98 image remove with multiple names, by name
ok 99 image remove with multiple names, by ID
ok 100 image volume ignore
ok 101 image volume bind
ok 102 image volume user mkdir
ok 103 info inspect
ok 104 ctr inspect
ok 105 ctr inspect not found
ok 106 inspect image should succed contain all necessary information
ok 107 pid_namespace_mode_pod_test
ok 108 ensure correct hostname
ok 109 ensure correct hostname for hostnetwork:true
ok 110 Check for valid pod netns CIDR
ok 111 Ensure correct CNI plugin namespace/name/container-id arguments
ok 112 Connect to pod hostport from the host
ok 113 Clean up network if pod sandbox fails
ok 114 Clean up network if pod sandbox fails after plugin success
ok 115 Ping pod from the host
ok 116 Ping pod from another pod
ok 117 pod release name on remove
ok 118 pod remove
ok 119 pod stop ignores not found sandboxes
ok 120 pod list filtering
ok 121 pod metadata in list & status
ok 122 pass pod sysctls to runtime
ok 123 pod stop idempotent
ok 124 pod remove idempotent
ok 125 pod stop idempotent with ctrs already stopped
ok 126 restart crio and still get pod status
ok 127 invalid systemd cgroup_parent fail
ok 128 systemd cgroup_parent correctly set
ok 129 kubernetes pod terminationGracePeriod passthru
ok 130 pod pause image matches configured image in crio.conf
ok 131 pprof
ok 132 reload config should succeed
ok 133 reload config should succeed with 'log_level'
ok 134 reload config should fail with 'log_level' if invalid
ok 135 reload config should fail with if config is malformed
ok 136 reload config should succeed with 'pause_image'
ok 137 reload config should succeed with 'pause_command'
ok 138 reload config should succeed with 'pause_image_auth_file'
ok 139 reload config should fail with non existing 'pause_image_auth_file'
ok 140 reload config should succeed with 'log_filter'
ok 141 reload config should fail with invalid 'log_filter'
ok 142 reload config should succeed with 'decryption_keys_path'
ok 143 reload config should succeed with 'seccomp_profile'
ok 144 reload config should fail with invalid 'seccomp_profile'
ok 145 reload config should succeed with 'apparmor_profile' # skip skip test since AppArmor is not enabled.
ok 146 reload config should fail with invalid 'apparmor_profile' # skip skip test since AppArmor is not enabled.
ok 147 reload system registries should succeed
ok 148 reload system registries should succeed with new registry
ok 149 reload system registries should fail on invalid syntax in file
ok 150 crio restore
ok 151 crio restore with bad state and pod stopped
ok 152 crio restore with bad state and ctr stopped
ok 153 crio restore with bad state and ctr removed
ok 154 crio restore with bad state and pod removed
ok 155 crio restore with bad state
ok 156 crio restore with missing config.json
ok 157 crio restore first not managing then managing
ok 158 crio restore first managing then not managing
ok 159 crio restore changing managing dir
ok 160 crictl runtimeversion
ok 161 if fs.may_detach_mounts is set
ok 162 ctr termination reason Completed
ok 163 ctr check shared /dev/shm
ok 164 stats
ok 165 container stats
ok 166 status should fail if no subcommand is provided
ok 167 status should succeed to retrieve the config
ok 168 status should fail to retrieve the config with invalid socket
ok 169 status should succeed to retrieve the info
ok 170 status should fail to retrieve the info with invalid socket
ok 171 succeed to retrieve the container info
ok 172 should fail to retrieve the container info without ID
ok 173 should fail to retrieve the container with invalid socket
ok 174 version should succeed
ok 175 version should succeed with JSON

real\t6m33.342s
user\t13m26.545s
sys\t5m9.728s"]}

@kolyshkin
Copy link
Collaborator Author

kolyshkin commented Feb 26, 2021

There's obviously a problem with prow, as the test logs are not appearing in

https://prow.ci.openshift.org/view/gcs/origin-federated-results/pr-logs/pull/cri-o_cri-o/4605/test_pull_request_crio_integration_crun_fedora/14004/

The logs are failed to upload, according to https://prow.ci.openshift.org/log?job=test_pull_request_crio_integration_crun_fedora&id=1365086716376911880

{"component":"gcsupload","file":"prow/gcsupload/run.go:192","func":"k8s.io/test-infra/prow/gcsupload.gatherArtifacts.func1","level":"info","msg":"Found /data/gcs/artifacts/testout.txt in artifact directory. Uploading as pr-logs/pull/cri-o_cri-o/4605/test_pull_request_crio_integration_crun_fedora/14004/artifacts/testout.txt\n","severity":"info","time":"2021-02-26T00:11:48Z"}
...
{"component":"gcsupload","dest":"pr-logs/pull/cri-o_cri-o/4605/test_pull_request_crio_integration_crun_fedora/14004/artifacts/testout.txt","file":"prow/pod-utils/gcs/upload.go:88","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2021-02-26T00:11:48Z"}
...
{"component":"gcsupload","dest":"pr-logs/pull/cri-o_cri-o/4605/test_pull_request_crio_integration_crun_fedora/14004/artifacts/testout.txt","file":"prow/pod-utils/gcs/upload.go:114","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Failed upload","severity":"info","time":"2021-02-26T00:31:09Z"}

Not sure what to do with it.

@openshift-ci-robot openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 28, 2021
@kolyshkin kolyshkin closed this Nov 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: no Indicates the PR's author has not DCO signed all their commits. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants