stats: modernize stats collection and implement Pod level stats reporting #5427

haircommander · 2021-10-26T22:05:00Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

CRI stats has been in sore need of an update, and is targeted to be updated in 1.23. Part of this update is support in the main CRI implementations.

This PR is partially a refactor and partially a feature. In includes cleaning up some code that wasn't actually used and was thus wasteful, bumping CRI to get the new pod level stats calls, adding a StatsServer to give the ability to cache the stats on a periodic basis (rather than collect on-demand, as is default).

There still should be some test updates, but integration tests will be hard before critools adds support for the new fields and cuts a release. In the meantime, lots of this is reviewable.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Add a [crio.stats] table to the config for configuring all things stats-related.
Add ability (with config field StatsCollectionPeriod) to calculate and save stats for pods and containers, rather than return them on demand.
Add support for {,List}PodSandboxStats, which allows the kubelet to take advantage of the PodAndContainerStatsFromCRI feature gate--one that is set to be alpha in 1.23.

codecov · 2021-10-26T22:11:39Z

Codecov Report

Merging #5427 (9dedd51) into main (31faba4) will decrease coverage by 0.06%.
The diff coverage is 29.84%.

❗ Current head 9dedd51 differs from pull request most recent head dec3bf5. Consider uploading reports for the commit dec3bf5 to get more accurate results

@@            Coverage Diff             @@
##             main    #5427      +/-   ##
==========================================
- Coverage   43.52%   43.45%   -0.07%     
==========================================
  Files         118      120       +2     
  Lines       11890    11924      +34     
==========================================
+ Hits         5175     5182       +7     
- Misses       6220     6245      +25     
- Partials      495      497       +2

sonatype-lift · 2021-10-26T22:34:04Z

internal/config/cgmgr/stats.go

+
+func UpdateWithMemoryStatsFromFile(memoryStatPath, inactiveFileSearchString string, memory *types.MemoryUsage, usage uint64) error {
+	var totalInactive uint64
+	f, err := os.Open(memoryStatPath)


G304: Potential file inclusion via variable
(at-me in a reply with help or ignore)

haircommander · 2021-10-29T14:06:43Z

/test integration_fedora

haircommander · 2021-10-29T15:01:02Z

/retest

saschagrunert

Just a few nits, otherwise LGTM

internal/config/cgmgr/stats.go

fidencio · 2021-11-08T08:17:44Z

@haircommander, can you hold this one till Tomorrow? I'd like to do a review (together with some local tests) on this later Today / Tomorrow morning.

fidencio · 2021-11-14T08:33:46Z

internal/oci/runtime_vm.go

+		rssBytes = m.Memory.RSS
+		pageFaults = m.Memory.PgFault
+		majorPageFaults = m.Memory.PgMajFault


For kata-containers, this is the part that differs from the approach taken before and that we should ensure the CRI runtime has access to the correct information.

Those are collect, and then passed up to the CRI runtime, here:
https://github.com/kata-containers/kata-containers/blob/e89c06e68bcd465cc79dc68945751a706f230bfe/src/runtime/pkg/containerd-shim-v2/metrics.go#L98-L146

It means that, theoretically, from the kata-containers everything needed here is already provided by kata-containers, but I still need to get to do some more tests on this.

internal/oci/runtime_vm.go

haircommander · 2021-11-22T20:54:42Z

comments addressed, PTAL @cri-o/cri-o-maintainers

saschagrunert · 2021-11-23T08:10:21Z

Unfortunately this needs a rebase.

Signed-off-by: Kir Kolyshkin <[email protected]>

Signed-off-by: Peter Hunt <[email protected]>

haircommander · 2021-11-23T14:23:53Z

rebased!

- drop unneeded fields from stats call - remove unneeded variable - replace internal Metadata with CRI type - as well as add CRIAttributes function - update with most remaining required fields for container stats Signed-off-by: Peter Hunt <[email protected]>

Signed-off-by: Peter Hunt <[email protected]>

saschagrunert

/lgtm

openshift-ci · 2021-11-23T15:52:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, kolyshkin, saschagrunert

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [haircommander,kolyshkin,saschagrunert]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

haircommander · 2021-11-23T16:49:54Z

/retest

openshift-ci · 2021-11-23T17:53:52Z

@haircommander: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/openshift-jenkins/e2e_crun_cgroupv2	`dec3bf5`	link	false	`/test e2e_cgroupv2`
ci/openshift-jenkins/integration_crun_cgroupv2	`dec3bf5`	link	false	`/test integration_cgroupv2`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

haircommander · 2021-11-23T17:58:00Z

/retest-required

openshift-bot · 2021-11-23T18:03:49Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

haircommander requested review from fidencio, mrunalp and runcom as code owners October 26, 2021 22:05

openshift-ci bot added kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Oct 26, 2021

openshift-ci bot requested a review from kolyshkin October 26, 2021 22:05

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 26, 2021

haircommander force-pushed the cri-stats-enhancement branch from 3a4b9fe to cfe818d Compare October 26, 2021 22:07

sonatype-lift bot reviewed Oct 26, 2021

View reviewed changes

haircommander force-pushed the cri-stats-enhancement branch 5 times, most recently from 31e8e0b to 3644f70 Compare October 28, 2021 13:48

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2021

haircommander force-pushed the cri-stats-enhancement branch from 3644f70 to 8f78f4f Compare October 28, 2021 13:54

openshift-ci bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2021

haircommander force-pushed the cri-stats-enhancement branch from 8f78f4f to 1402213 Compare October 28, 2021 19:22

haircommander added this to the 1.23 milestone Nov 5, 2021

saschagrunert approved these changes Nov 8, 2021

View reviewed changes

internal/config/cgmgr/stats.go Outdated Show resolved Hide resolved

internal/config/cgmgr/stats.go Outdated Show resolved Hide resolved

internal/config/cgmgr/stats.go Outdated Show resolved Hide resolved

haircommander force-pushed the cri-stats-enhancement branch from 1402213 to ebd34ac Compare November 10, 2021 22:19

fidencio reviewed Nov 14, 2021

View reviewed changes

internal/oci/runtime_vm.go Show resolved Hide resolved

openshift-ci bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 17, 2021

haircommander force-pushed the cri-stats-enhancement branch from ebd34ac to e4bb4e6 Compare November 18, 2021 17:03

haircommander force-pushed the cri-stats-enhancement branch from 6e0436c to 5fec39a Compare November 22, 2021 20:40

kolyshkin and others added 5 commits November 23, 2021 09:22

pkg/config: use iota

a598deb

Signed-off-by: Kir Kolyshkin <[email protected]>

server/cri: refactor to make stats processing unified

c5dd30d

Signed-off-by: Peter Hunt <[email protected]>

vendor: bump cri-api

ad71bd9

Signed-off-by: Peter Hunt <[email protected]>

server/cri: add PodSandboxStats support

542eb55

Signed-off-by: Peter Hunt <[email protected]>

server: stub {List,}PodSandboxStats

536c084

Signed-off-by: Peter Hunt <[email protected]>

haircommander force-pushed the cri-stats-enhancement branch from 5fec39a to e34f415 Compare November 23, 2021 14:23

haircommander added 8 commits November 23, 2021 09:24

cgmgr: move most of stats handling to cgmgr

2569255

Signed-off-by: Peter Hunt <[email protected]>

config: add StatsCollectionPeriod field

43db34f

Signed-off-by: Peter Hunt <[email protected]>

stats: add stats server

7f13683

Signed-off-by: Peter Hunt <[email protected]>

container server: use stats server

b17b7df

Signed-off-by: Peter Hunt <[email protected]>

server: use stats server to get container stats

64870e3

Signed-off-by: Peter Hunt <[email protected]>

server: refactor sandbox list

5ba5cb0

Signed-off-by: Peter Hunt <[email protected]>

server: add {,List}SandboxStats

dec3bf5

Signed-off-by: Peter Hunt <[email protected]>

haircommander force-pushed the cri-stats-enhancement branch from e34f415 to dec3bf5 Compare November 23, 2021 14:24

saschagrunert approved these changes Nov 23, 2021

View reviewed changes

openshift-ci bot assigned saschagrunert Nov 23, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 23, 2021

openshift-merge-robot merged commit 036030d into cri-o:main Nov 23, 2021

saschagrunert mentioned this pull request Nov 29, 2021

Fix runtime panic on stats server shutdown #5474

Merged

HirazawaUi mentioned this pull request Jul 31, 2024

Allow retrieving container stats from cri when using crio kubernetes/kubernetes#126487

Closed

stats: modernize stats collection and implement Pod level stats reporting #5427

stats: modernize stats collection and implement Pod level stats reporting #5427

Uh oh!

Conversation

haircommander commented Oct 26, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

codecov bot commented Oct 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonatype-lift bot Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

haircommander commented Oct 29, 2021

Uh oh!

haircommander commented Oct 29, 2021

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fidencio commented Nov 8, 2021

Uh oh!

fidencio Nov 14, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

haircommander commented Nov 22, 2021

Uh oh!

saschagrunert commented Nov 23, 2021

Uh oh!

haircommander commented Nov 23, 2021

Uh oh!

saschagrunert left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Nov 23, 2021

Uh oh!

haircommander commented Nov 23, 2021

Uh oh!

openshift-ci bot commented Nov 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

haircommander commented Nov 23, 2021

Uh oh!

openshift-bot commented Nov 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented Oct 26, 2021 •

edited

Loading

openshift-ci bot commented Nov 23, 2021 •

edited

Loading