Fix bogus CI test failures #4217

kolyshkin · 2020-09-21T19:39:00Z

/kind failing-test

What this PR does / why we need it:

The following failure happens in CI on RHEL7 from time to time:

removing the pod sandbox "0d5bf5eeb0048bb70ab8ee9bca0a497e216a6c6cfa42507a8d735f61c825784d": rpc error: code = Unknown desc = unable to remove managed namespaces: Removing namespaces encountered the following errors [unlinkat /var/run/netns/0376543d-4917-417f-b406-7e8ab07cb847: device or resource busy]

(with different test cases and different namespaces).

RHEL7 kernel has upstream commit [1] backported, but the feature is
controlled by a sysctl and is off by default. For the feature to work,
one needs to set fs.may_detach_mounts = 1.

On production RHEL7 systems, this is done by runc rpm (see [2]), but
we're not using those rpms for CI, so we have to set the sysctl
manually.

Add an integration test case to check the sysctl is set.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1823374#c17

Which issue(s) this PR fixes:

Fixes: #3996
Some more notes at #4210

Special notes for your reviewer:

Does this PR introduce a user-facing change?

None

haircommander · 2020-09-21T19:47:12Z

test/sanity_checks.bats

@@ -0,0 +1,20 @@
+#!/usr/bin/env bats


I think more users are apt to find this out if it's in cri-o itself. we could have a log message come out of internal/config/node similarly to how we check other node level knobs. wdyt

This one is strictly for CI, so it would fail in case we are not running in a sane env.

Adding a log message to cri-o itself is also on my radar. Added.

Not sure if we should make it fatal or not. I'd rather make it fatal as otherwise we'll have to deal with some weird EBUSY bugs for no reason.

contrib/test/integration/main.yml

codecov · 2020-09-21T20:02:03Z

Codecov Report

Merging #4217 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4217   +/-   ##
=======================================
  Coverage   38.72%   38.72%           
=======================================
  Files         111      111           
  Lines        8702     8702           
=======================================
  Hits         3370     3370           
  Misses       4967     4967           
  Partials      365      365

haircommander · 2020-09-21T20:32:16Z

/test ami

(I'm not sure the ami update jobs actually work anymore, but it's worth a shot)

kolyshkin · 2020-09-21T20:59:01Z

Apparently it's working; from integration_rhel output:

ok 163 if fs.may_detach_mounts is set

kolyshkin · 2020-09-21T22:37:44Z

A bunch of failed CI jobs doesn't have any logs...

/retest

kolyshkin · 2020-09-22T00:36:54Z

A minor fix to the error message...

mrunalp · 2020-09-22T21:44:08Z

From e2e_fedora logs:

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to reload sysctl: net.ipv4.conf.all.route_localnet = 1\nnet.ipv4.ping_group_range = 0 2147483647\nnet.bridge.bridge-nf-call-iptables = 1\nsysctl: cannot stat /proc/sys/fs/may_detach_mounts: No such file or directory\n"}

kolyshkin · 2020-09-28T14:45:57Z

contrib/test/integration/main.yml

+        state: present
+        value: 1
+        sysctl_set: yes
+        ignoreerrors: yes


ignoreerrors boolean
Choices:
no ←
yes
Use this option to ignore errors about unknown keys.

Alas, this is not working for some reason

Maybe it's because of state: present stance. Let me try removing it.

retrying with state: absent.

The following failure happens in CI on RHEL7 from time to time: > removing the pod sandbox "0d5bf5eeb0048bb70ab8ee9bca0a497e216a6c6cfa42507a8d735f61c825784d": rpc error: code = Unknown desc = unable to remove managed namespaces: Removing namespaces encountered the following errors [unlinkat /var/run/netns/0376543d-4917-417f-b406-7e8ab07cb847: device or resource busy] (with different test cases and different namespaces). RHEL7 kernel has upstream commit [1] backported, but the feature is controlled by a sysctl and is off by default. For the feature to work, one needs to set fs.may_detach_mounts = 1. On production RHEL7 systems, this is done by runc rpm (see [2]), but we're not using those rpms for CI, so we have to set the sysctl manually. Add an integration test case to check the sysctl is set. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe [2] https://bugzilla.redhat.com/show_bug.cgi?id=1823374#c17 Signed-off-by: Kir Kolyshkin <[email protected]>

kolyshkin · 2020-09-28T20:48:02Z

internal/config/node/node.go

+			init:      checkFsMayDetachMounts,
+			err:       &checkFsMayDetachMountsErr,
+			activated: nil,
+			fatal:     true,


The only question is whether to make it fatal. I'm in favor to do so, and this is gross misconfiguration which will result in various failures later, so it's better to bail out.

Basically if we fail here it means RPM packaging is broken or something like that.

internal/config/node/sysctl.go

... and make it fatal. See previous commit for details. Signed-off-by: Kir Kolyshkin <[email protected]>

openshift-ci-robot · 2020-09-28T21:56:11Z

@kolyshkin: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/ami_fedora	`58020e4`	link	`/test ami_fedora`
ci/openshift-jenkins/ami_rhel	`58020e4`	link	`/test ami_rhel`

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

kolyshkin · 2020-09-28T22:18:20Z

ci/openshift-jenkins/integration_crun_cgroupv2 — Jenkins job failed.

and if you click on "Details" link it says

Test started today at 2:36 PM is still running after 39m15s. (more info)

:(

kolyshkin · 2020-09-28T23:51:06Z

/retest

kolyshkin · 2020-09-29T00:48:31Z

Look ma, green CI 🎉

mrunalp · 2020-09-29T00:49:39Z

/lgtm

openshift-ci-robot · 2020-09-29T00:50:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kolyshkin, mrunalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kolyshkin · 2020-09-29T18:13:03Z

/cherry-pick release-1.19

openshift-cherrypick-robot · 2020-09-29T18:13:18Z

@kolyshkin: new pull request created: #4228

Details

In response to this:

/cherry-pick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kolyshkin · 2020-09-29T18:28:24Z

/cherry-pick release-1.18

openshift-cherrypick-robot · 2020-09-29T18:28:28Z

@kolyshkin: #4217 failed to apply on top of branch "release-1.18":

Applying: Fix bogus CI test failures
Applying: internal/config/node: add checkFsMayDetachMounts
Using index info to reconstruct a base tree...
A	internal/config/node/node.go
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): internal/config/node/node.go deleted in HEAD and modified in internal/config/node: add checkFsMayDetachMounts. Version internal/config/node: add checkFsMayDetachMounts of internal/config/node/node.go left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0002 internal/config/node: add checkFsMayDetachMounts
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-1.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kolyshkin · 2020-09-29T18:47:32Z

Backports:

kolyshkin requested review from mrunalp and runcom as code owners September 21, 2020 19:39

openshift-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Sep 21, 2020

openshift-ci-robot requested a review from sameo September 21, 2020 19:39

haircommander reviewed Sep 21, 2020

View reviewed changes

mrunalp reviewed Sep 21, 2020

View reviewed changes

contrib/test/integration/main.yml Outdated Show resolved Hide resolved

kolyshkin force-pushed the fs-may-detach-mounts branch from 58020e4 to 9819474 Compare September 21, 2020 21:01

kolyshkin force-pushed the fs-may-detach-mounts branch from 06dc9ee to 721ed24 Compare September 22, 2020 00:36

kolyshkin commented Sep 28, 2020

View reviewed changes

kolyshkin force-pushed the fs-may-detach-mounts branch from 721ed24 to 25b8c02 Compare September 28, 2020 15:14

kolyshkin commented Sep 28, 2020

View reviewed changes

kolyshkin mentioned this pull request Sep 28, 2020

Fix make vendor #4208

Merged

kolyshkin requested review from haircommander and mrunalp September 28, 2020 20:51

mrunalp reviewed Sep 28, 2020

View reviewed changes

internal/config/node/sysctl.go Outdated Show resolved Hide resolved

internal/config/node: add checkFsMayDetachMounts

bd3aa81

... and make it fatal. See previous commit for details. Signed-off-by: Kir Kolyshkin <[email protected]>

kolyshkin force-pushed the fs-may-detach-mounts branch from 25b8c02 to bd3aa81 Compare September 28, 2020 21:34

openshift-ci-robot assigned mrunalp Sep 29, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 29, 2020

mrunalp approved these changes Sep 29, 2020

View reviewed changes

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 29, 2020

openshift-merge-robot merged commit 1455ee8 into cri-o:master Sep 29, 2020

openshift-cherrypick-robot mentioned this pull request Sep 29, 2020

[release-1.19] Fix bogus CI test failures #4228

Merged

This was referenced Sep 29, 2020

[1.17] Fix bogus CI test failures on RHEL7 #4231

Merged

[1.18] Fix bogus CI test failures on RHEL7 #4232

Merged

This was referenced Feb 13, 2021

config/node: allow CRI-O to start if fs_may_detach_mounts is not set #4559

Closed

CRI-O service is not restarting after restart of the node #4460

Closed

Fix bogus CI test failures #4217

Fix bogus CI test failures #4217

Uh oh!

Conversation

kolyshkin commented Sep 21, 2020

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Sep 21, 2020

Codecov Report

Uh oh!

haircommander commented Sep 21, 2020

Uh oh!

kolyshkin commented Sep 21, 2020

Uh oh!

kolyshkin commented Sep 21, 2020

Uh oh!

kolyshkin commented Sep 22, 2020

Uh oh!

mrunalp commented Sep 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

openshift-ci-robot commented Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kolyshkin commented Sep 28, 2020

Uh oh!

kolyshkin commented Sep 28, 2020

Uh oh!

kolyshkin commented Sep 29, 2020

Uh oh!

mrunalp commented Sep 29, 2020

Uh oh!

openshift-ci-robot commented Sep 29, 2020

Uh oh!

kolyshkin commented Sep 29, 2020

Uh oh!

openshift-cherrypick-robot commented Sep 29, 2020

Uh oh!

kolyshkin commented Sep 29, 2020

Uh oh!

openshift-cherrypick-robot commented Sep 29, 2020

Uh oh!

kolyshkin commented Sep 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

openshift-ci-robot commented Sep 28, 2020 •

edited

Loading