Add GuestAgentInfo load metrics #14879

machadovilaca · 2025-06-06T10:33:55Z

What this PR does

Before this PR:

No GuestAgentInfo metrics reported

After this PR:

GuestAgentInfo cpu load metrics reported

# HELP kubevirt_vmi_guest_load_15m Guest system load average over 15 minutes as reported by the guest agent. Load is defined as the number of processes in the runqueue or waiting for disk I/O.
# TYPE kubevirt_vmi_guest_load_15m gauge
kubevirt_vmi_guest_load_15m{kubernetes_vmi_label_kubevirt_io_domain="vm1",kubernetes_vmi_label_kubevirt_io_nodeName="minikube",name="vm1",namespace="default",node="minikube"} 0

# HELP kubevirt_vmi_guest_load_1m Guest system load average over 1 minute as reported by the guest agent. Load is defined as the number of processes in the runqueue or waiting for disk I/O.
# TYPE kubevirt_vmi_guest_load_1m gauge
kubevirt_vmi_guest_load_1m{kubernetes_vmi_label_kubevirt_io_domain="vm1",kubernetes_vmi_label_kubevirt_io_nodeName="minikube",name="vm1",namespace="default",node="minikube"} 0.0029296875

# HELP kubevirt_vmi_guest_load_5m Guest system load average over 5 minutes as reported by the guest agent. Load is defined as the number of processes in the runqueue or waiting for disk I/O.
# TYPE kubevirt_vmi_guest_load_5m gauge
kubevirt_vmi_guest_load_5m{kubernetes_vmi_label_kubevirt_io_domain="vm1",kubernetes_vmi_label_kubevirt_io_nodeName="minikube",name="vm1",namespace="default",node="minikube"} 0.01611328125

Fixes #

kubevirt/enhancements#67

jira-ticket: https://issues.redhat.com/browse/CNV-50883

References

Why we need it and why it was done in this way

The following tradeoffs were made:

The following alternatives were considered:

Links to places where the discussion took place:

Special notes for your reviewer

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

Design: A design document was considered and is present (link) or not required
PR: The PR description is expressive enough and will help future contributors
Code: Write code that humans can understand and Keep it simple
Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
Upgrade: Impact of this change on upgrade flows was considered and addressed if required
Testing: New code requires new unit tests. New features and bug fixes require at least one e2e test
Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
Community: Announcement to kubevirt-dev was considered

Release note

Add GuestAgentInfo info metrics

sourcery-ai

Hey @machadovilaca - I've reviewed your changes and they look great!

Here's what I looked at during the review

🟡 General issues: 2 issues found
🟢 Security: all looks good
🟢 Testing: all looks good
🟢 Complexity: all looks good
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

staging/src/kubevirt.io/client-go/api/openapi_generated.go

pkg/monitoring/metrics/virt-handler/domainstats/guest_metrics.go

machadovilaca · 2025-06-16T10:10:20Z

/cc @enp0s3 @sradco

sradco · 2025-06-16T10:47:34Z

Hi, Where did this request came from?
Is there a Jira item?

sradco · 2025-06-16T10:48:54Z

docs/observability/metrics.md

+Guest hostname reported by the guest agent. The value is always 1. Type: Gauge.
+
+### kubevirt_vmi_guest_load_15m
+Guest system load average over 15 minutes as reported by the guest agent. Type: Gauge.


What does 'load' mean in this case?

average length of the cpu queue over the period of time

Please update this info in the metrics description

guest system load is the more accurate description as used the kernel

machadovilaca · 2025-06-16T11:33:55Z

Hi, Where did this request came from? Is there a Jira item?

hidden in the comment section of the PR description again, moved out

sradco · 2025-06-17T20:14:37Z

docs/observability/metrics.md

 ### kubevirt_vmi_filesystem_used_bytes
 Used VM filesystem capacity in bytes. Type: Gauge.

+### kubevirt_vmi_guest_hostname_info


Why should this be a separate metric and not add it to the kubevirt_vm/i_info metrics?

enp0s3 · 2025-07-21T16:38:05Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_collector.go

+		}
+
+		guestInfo := vmiReport.vmiStats.GuestAgentInfo
+


Here we also need to check whether guestInfo.Load isn't nil since its a pointer:

// Load contains the system load averages (1M, 5M, 15M) from the guest agent Load *VirtualMachineInstanceGuestOSLoad `json:"load,omitempty"` }

https://github.com/kubevirt/kubevirt/pull/14879/files#diff-49a70564a161bb7fb780be9033d25912b2bffb34436d51d6e2adbd01413fe71aR87

L87 checks if vmiReport.vmiStats.GuestAgentInfo is not nil, but it doesn't check if vmiReport.vmiStats.GuestAgentInfo.Load is not nil.

you are right, missed it, updated

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper.go

enp0s3 · 2025-07-28T11:01:11Z

/approve
/hold

Lets add the tests and fix the Load field validation, then feel free to unhold

kubevirt-bot · 2025-07-28T11:01:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enp0s3

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [enp0s3]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

0xFelix · 2025-07-28T12:46:43Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+	})
+
+	Context("Deep copy behavior", func() {
+		It("should store a deep copy of guest info in cache", func() {


Not sure about the value of this test. DeepCopy is generated could be tested without going through the cache but I don't think we need to do that.

0xFelix · 2025-07-28T12:47:32Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+			scraper.mutex.Lock()
+			scraper.cache[socketFile] = &guestAgentInfoCache{
+				timestamp: time.Now(),
+				info:      guestInfo.DeepCopy(),
+			}
+			scraper.mutex.Unlock()


nit: You could extract blocks like this using the mutexes into helpers

0xFelix

Are the failures in lane sig-compute related?

0xFelix · 2025-07-29T15:11:11Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_collector.go

+		}
+
+		guestInfo := vmiReport.vmiStats.GuestAgentInfo
+


L87 checks if vmiReport.vmiStats.GuestAgentInfo is not nil, but it doesn't check if vmiReport.vmiStats.GuestAgentInfo.Load is not nil.

0xFelix · 2025-07-29T15:13:24Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+
+			By("checking if entry is within timeout")
+			cached, exists := getCacheEntry(socketFile)
+			withinTimeout := exists && time.Since(cached.timestamp) < cacheTimeout


This does not verify controller logic? I'd like to see a test that verifies that the controller doesn't update if the cached value is still within the timeout range.

i understand what you mean, it was making the requests to the scrapper cache
but the validation had to be local because of the way the functions were built
i refactored the scrapper to allow us to test each function individually

0xFelix · 2025-07-29T15:14:13Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+
+			By("checking if entry is outside timeout")
+			cached, exists := getCacheEntry(socketFile)
+			withinTimeout := exists && time.Since(cached.timestamp) < cacheTimeout


Same, this doesn't test controller logic, but rather a condition in the test code. Was this generated by AI?

#14879 (comment)

0xFelix · 2025-07-29T15:15:22Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+		It("should not clean up non-expired cache entries", func() {
+			By("adding a fresh cache entry")
+			addCacheEntry(socketFile, time.Now(), guestInfo)
+
+			By("calling cleanup immediately")
+			scraper.cleanupExpiredCache()
+
+			By("verifying cache entry still exists")
+			exists := cacheEntryExists(socketFile)
+			Expect(exists).To(BeTrue())
+		})


This case is a duplicate of the case above?

pkg/virt-launcher/virtwrap/agent-poller/agent_poller.go

0xFelix · 2025-07-29T15:16:58Z

staging/src/kubevirt.io/api/core/v1/types.go

+	// Load average over 1 minute
+	Load1m float64 `json:"load1m,omitempty"`
+	// Load5mSet indicates whether the 5 minute load average is set
+	Load5mSet bool `json:"load5mSet,omitempty"`


What does set mean? Why can't the actual field be a ptr instead?

we use the same format as libvirt.DomainGuestInfoLoad
so they set it like that

we could still make the conversion to pointer
but in every other domainstat where this is the case, we've been using the set
so I think should be better to keep it consistent

Can you point to an example?

pkg/virt-launcher/virtwrap/stats/types.go

That's internal API though. Which brings me back to I don't think we should litter the external API too much. Didn't we want to add this to the metrics endpoint provided by virt-launcher?

I see, this struct isn't used in the VMI's status.

machadovilaca · 2025-07-31T08:28:51Z

/retest

machadovilaca · 2025-07-31T08:29:13Z

Are the failures in lane sig-compute related?

@0xFelix it was related, fixed

0xFelix · 2025-07-31T09:00:49Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper.go

+func (d *GuestAgentInfoScraper) getCacheEntry(socketFile string) (*VirtualMachineInstanceStats, time.Time, bool) {
+	now := time.Now()
+
+	d.mutex.RLock()


You call RLock but then you write to d.cache?

true, it does not make sense to have a RW mutex I think

0xFelix · 2025-07-31T09:02:32Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper_test.go

+			By("checking if entry is outside timeout")
+			_, timestamp, exists := scraper.getCacheEntry(socketFile)
+			Expect(exists).To(BeFalse())
+			Expect(time.Since(timestamp)).To(BeNumerically(">", cacheTimeout))


timestamp should be an empty time.Time{}?

can you clarify?

got it, updated

0xFelix · 2025-07-31T09:03:01Z

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper.go

+	if vmStats.GuestAgentInfo != nil && vmStats.GuestAgentInfo.Hostname != "" {
+		d.addCacheEntry(socketFile, time.Now(), vmStats.GuestAgentInfo)
+	}


Can you add a unit test for this behavior?

pkg/virt-launcher/virtwrap/agent-poller/agent_poller.go

0xFelix · 2025-07-31T09:05:30Z

staging/src/kubevirt.io/api/core/v1/types.go

+	// Load average over 1 minute
+	Load1m float64 `json:"load1m,omitempty"`
+	// Load5mSet indicates whether the 5 minute load average is set
+	Load5mSet bool `json:"load5mSet,omitempty"`


Can you point to an example?

Signed-off-by: João Vilaça <[email protected]>

kubevirt-bot · 2025-07-31T14:47:37Z

@machadovilaca: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubevirt-verify-rpms	`92b67bb`	link	false	`/test pull-kubevirt-verify-rpms`
pull-kubevirt-check-tests-for-flakes	`1f0dfc1`	link	false	`/test pull-kubevirt-check-tests-for-flakes`

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

0xFelix

/lgtm

kubevirt-commenter-bot · 2025-08-01T13:43:21Z

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-k8s-1.31-windows2016
/test pull-kubevirt-e2e-kind-1.33-vgpu
/test pull-kubevirt-e2e-kind-sriov
/test pull-kubevirt-e2e-k8s-1.33-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.31-sig-network
/test pull-kubevirt-e2e-k8s-1.31-sig-storage
/test pull-kubevirt-e2e-k8s-1.31-sig-compute
/test pull-kubevirt-e2e-k8s-1.31-sig-operator
/test pull-kubevirt-e2e-k8s-1.32-sig-network
/test pull-kubevirt-e2e-k8s-1.32-sig-storage
/test pull-kubevirt-e2e-k8s-1.32-sig-compute
/test pull-kubevirt-e2e-k8s-1.32-sig-operator

machadovilaca · 2025-08-01T13:43:42Z

/unhold

/cc @enp0s3

kubevirt-commenter-bot · 2025-08-01T15:55:34Z

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

kubevirt-commenter-bot · 2025-08-01T18:55:34Z

/retest-required
This bot automatically retries required jobs that failed/flaked on approved PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 6, 2025

kubevirt-bot requested review from Barakmor1, avlitman, brianmcarey and fossedihelm June 6, 2025 10:34

kubevirt-bot added the area/launcher label Jun 6, 2025

kubevirt-bot requested review from nunnatsa and victortoso June 6, 2025 10:34

dosubot bot added the area/monitoring label Jun 6, 2025

sourcery-ai bot reviewed Jun 6, 2025

View reviewed changes

staging/src/kubevirt.io/client-go/api/openapi_generated.go Show resolved Hide resolved

pkg/monitoring/metrics/virt-handler/domainstats/guest_metrics.go Outdated Show resolved Hide resolved

machadovilaca force-pushed the add-guestloadinfo branch 2 times, most recently from 02231c6 to 9b1e901 Compare June 9, 2025 09:18

kubevirt-bot requested review from enp0s3 and sradco June 16, 2025 10:10

sradco reviewed Jun 16, 2025

View reviewed changes

sradco approved these changes Jun 17, 2025

View reviewed changes

kubevirt-bot assigned sradco Jun 17, 2025

kubevirt-bot added lgtm Indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jun 17, 2025

machadovilaca force-pushed the add-guestloadinfo branch from 9b1e901 to f348447 Compare June 20, 2025 13:14

kubevirt-bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 20, 2025

kubevirt-bot added the size/XL label Jul 15, 2025

enp0s3 reviewed Jul 21, 2025

View reviewed changes

pkg/monitoring/metrics/virt-handler/domainstats/guest_agent_info_scraper.go Show resolved Hide resolved

kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 28, 2025

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 28, 2025

0xFelix reviewed Jul 28, 2025

View reviewed changes

machadovilaca force-pushed the add-guestloadinfo branch 2 times, most recently from 05987f1 to 9b02250 Compare July 28, 2025 13:24

0xFelix reviewed Jul 29, 2025

View reviewed changes

machadovilaca force-pushed the add-guestloadinfo branch 2 times, most recently from 566d98f to 9d407d7 Compare July 30, 2025 17:49

0xFelix reviewed Jul 31, 2025

View reviewed changes

machadovilaca force-pushed the add-guestloadinfo branch from 9d407d7 to a2524fd Compare July 31, 2025 10:42

machadovilaca mentioned this pull request Jul 31, 2025

Refactor code in pkg/virt-launcher/virtwrap/agent-poller/agent_poller.go #15342

Closed

Add GuestAgentInfo load metrics

1f0dfc1

Signed-off-by: João Vilaça <[email protected]>

machadovilaca force-pushed the add-guestloadinfo branch from a2524fd to 1f0dfc1 Compare July 31, 2025 12:38

0xFelix reviewed Aug 1, 2025

View reviewed changes

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 1, 2025

kubevirt-bot requested a review from enp0s3 August 1, 2025 13:43

kubevirt-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 1, 2025

kubevirt-bot merged commit 544a8a2 into kubevirt:main Aug 1, 2025
41 of 42 checks passed

Add GuestAgentInfo load metrics #14879

Add GuestAgentInfo load metrics #14879

Uh oh!

Conversation

machadovilaca commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

Before this PR:

After this PR:

References

Why we need it and why it was done in this way

Special notes for your reviewer

Checklist

Release note

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

machadovilaca commented Jun 16, 2025

Uh oh!

sradco commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machadovilaca commented Jun 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

enp0s3 commented Jul 28, 2025

Uh oh!

kubevirt-bot commented Jul 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

0xFelix left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

machadovilaca commented Jun 6, 2025 •

edited

Loading