Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@kolyshkin
Copy link
Collaborator

@kolyshkin kolyshkin commented Feb 18, 2022

/kind ci
/kind failing-test
/kind flake

What this PR does / why we need it:

This tries to solve a failure in building rpms, and consists of two fixes.

  1. Looking into cri-o CI, this error seems common:

http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2600:1f16:c1:5e03:a992:8ee1:99d2:6f8: Network is unreachable"
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2600:1f16:c1:5e03:a992:8ee1:99d2:6f8: Network is unreachable"
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] HTTPS Error 301 - Moved Permanently
[ERROR] hack/build-rpms.sh:31: yum-builddep -y "${OS_RPM_SPECFILE}" exited with status 1.

Apparently, vault.centos.org resolves to both IPv4 and IPv6 addresses,
and one of the addresses is used (randomly, I presume). If the node
does not have IPv6 connectivity (as it seems to be the case), as a
result we have 50% chance of failure.

Work around this by asking yum to only use IPv4 for address resolving.

  1. vault.centos.org is super slow, resulting in numerous timeouts, often fatal:

http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
[ERROR] hack/build-rpms.sh:38: yum-builddep -y "${OS_RPM_SPECFILE}" exited with status 1.

Let's use mirrors.kernel.org instead.

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

@openshift-ci openshift-ci bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/ci Categorizes issue or PR as related to CI labels Feb 18, 2022
@openshift-ci openshift-ci bot requested review from QiWang19 and klihub February 18, 2022 23:06
@openshift-ci openshift-ci bot added kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 18, 2022
@codecov
Copy link

codecov bot commented Feb 18, 2022

Codecov Report

Merging #5663 (6c3144a) into main (d0b00ab) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #5663      +/-   ##
==========================================
- Coverage   43.03%   43.02%   -0.01%     
==========================================
  Files         123      123              
  Lines       12286    12286              
==========================================
- Hits         5287     5286       -1     
- Misses       6491     6492       +1     
  Partials      508      508              

@kolyshkin

This comment was marked as outdated.

@kolyshkin
Copy link
Collaborator Author

OK, it kind of works now... producing a different error:

[INFO] Disabling IPv6 for yum ...
Loaded plugins: fastestmirror, ovl
Enabling base-source repository
Enabling epel-source repository
Enabling extras-source repository
Enabling updates-source repository
Determining fastest mirrors
 * base: mirror.mia.velocihost.net
 * epel: mirror.umd.edu
 * epel-source: mirror.umd.edu
 * extras: www.gtlib.gatech.edu
 * updates: mirror.atl.genesisadaptive.com
http://mirror.mia.velocihost.net/centos/7.9.2009/os/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on https://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation timed out after 30262 milliseconds with 0 out of 0 bytes received')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on https://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30344 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30000 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
failure: repodata/repomd.xml from base-source: [Errno 256] No more mirrors to try.
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on https://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation timed out after 30262 milliseconds with 0 out of 0 bytes received')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on https://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30344 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30000 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
[ERROR] hack/build-rpms.sh:38: `yum-builddep -y "${OS_RPM_SPECFILE}"` exited with status 1.

This tries to solve a failure in building rpms, and consists of two
fixes.

1. Looking into cri-o CI, this error seems common:

	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2600:1f16:c1:5e03:a992:8ee1:99d2:6f8: Network is unreachable"
	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] curl#7 - "Failed to connect to 2600:1f16:c1:5e03:a992:8ee1:99d2:6f8: Network is unreachable"
	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 14] HTTPS Error 301 - Moved Permanently
	[ERROR] hack/build-rpms.sh:31: `yum-builddep -y "${OS_RPM_SPECFILE}"` exited with status 1.

Apparently, vault.centos.org resolves to both IPv4 and IPv6 addresses,
and one of the addresses is used (randomly, I presume). If the node
does not have IPv6 connectivity (as it seems to be the case), as a
result we have 50% chance of failure.

Work around this by asking yum to only use IPv4 for address resolving.

2. vault.centos.org is super slow, resulting in numerous timeouts,
   often fatal:

	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
	http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: [Errno 12] Timeout on http://vault.centos.org/centos/7/os/Source/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')
	[ERROR] hack/build-rpms.sh:38: `yum-builddep -y "${OS_RPM_SPECFILE}"` exited with status 1.

Let's use mirrors.kernel.org instead.

Signed-off-by: Kir Kolyshkin <[email protected]>
@kolyshkin
Copy link
Collaborator Author

OK, this is centos 7 source repo, which is configured without mirrors, so yum has no alternative but to use vault.centos.org.

And vault.centos.org is super slow for some reason. I tried it locally and also see timeouts.

Let's try to use kernel.org mirror which has the sources, too.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 19, 2022

@kolyshkin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2 6c3144a link false /test e2e_cgroupv2

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kolyshkin
Copy link
Collaborator Author

All right, that finally works! @haircommander PTAL

@haircommander
Copy link
Member

amazing! thanks for digging into this @kolyshkin

/approve

@cri-o/cri-o-maintainers PTAL

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Feb 21, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, kolyshkin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [haircommander,kolyshkin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@TomSweeneyRedHat
Copy link
Contributor

Pretty slick @kolyshkin
LGTM

@haircommander
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 21, 2022
@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 7e5664d into cri-o:main Feb 21, 2022
@kolyshkin
Copy link
Collaborator Author

/cherry-pick release-1.23
/cherry-pick release-1.22

@openshift-cherrypick-robot

@kolyshkin: new pull request created: #5668

Details

In response to this:

/cherry-pick release-1.23
/cherry-pick release-1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@haircommander
Copy link
Member

/cherry-pick release-1.22

@haircommander
Copy link
Member

/cherry-pick release-1.21

@openshift-cherrypick-robot

@haircommander: new pull request could not be created: failed to create pull request against cri-o/cri-o#release-1.22 from head openshift-cherrypick-robot:cherry-pick-5663-to-release-1.22: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between cri-o:release-1.22 and openshift-cherrypick-robot:cherry-pick-5663-to-release-1.22"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

Details

In response to this:

/cherry-pick release-1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-cherrypick-robot

@haircommander: new pull request created: #5699

Details

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/ci Categorizes issue or PR as related to CI kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. kind/flake Categorizes issue or PR as related to a flaky test. lgtm Indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants