enable heap dumps #7328

hernandanielg · 2023-09-22T12:21:54Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This will help on the memory debugging process

Which issue(s) this PR fixes:

Fixes #7307

Special notes for your reviewer:

Does this PR introduce a user-facing change?

This PR will add a --enable-heap-dump flag to enable head dumps on an specific file

openshift-ci · 2023-09-22T12:22:00Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-09-22T12:22:10Z

Hi @hernandanielg. Thanks for your PR.

I'm waiting for a cri-o member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-09-22T12:22:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hernandanielg
Once this PR has been reviewed and has the lgtm label, please assign nalind for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sohankunkerkar · 2023-09-25T13:47:43Z

@hernandanielg Thanks for the PR.
You are on the right track. There are still a few places where you need to make changes. Did you see this PR to understand the actual logic? The idea is to dump the heap-related information if --enable-heap-dump is mentioned. Have you gone through this example PR to understand how to add a command line flag to the crio config?

sohankunkerkar · 2023-10-04T14:12:29Z

@hernandanielg are you still interested in pursuing this PR?

hernandanielg · 2023-10-04T18:02:53Z

yes! sorry I've been busy lately but will continue on this today 👋🏻

hernandanielg · 2023-10-24T16:42:26Z

hey @sohankunkerkar

As you may see I have added configuration option and reload function

I know it's just a matter to execute debug.WriteHeapDump(f.Fd()) as done here

I am trying to understand the code and find where and when it's going to be executed on runtime, maybe somewhere around here?

If you can guide me on this it would be highly appreciated!

Looking forward

sohankunkerkar · 2023-10-26T13:37:26Z

@hernandanielg thanks for the changes. I think you need to add the flag details to docs/ as well. As far as I know, you can keep the logic in server/inspect.go and call it only when that flag is enabled.

kwilczynski · 2023-11-06T02:22:31Z

pkg/config/reload.go


+// ReloadEnableHeapDump updates the DecryptionKeysPath with the provided
+// `newConfig`.
+func (c *Config) ReloadEnableHeapDump(newConfig *Config) {


@hernandanielg, wasn't this meant to be a command-line argument only?

The rationale behind this logic is to allow the user to change the location of the heap dump by reloading the config. So, we have to change the config option to a string type, rather than a boolean.

@sohankunkerkar, that makes sense. But, at the same time, what's wrong with using /tmp?

We aren't doing anything special for the goroutine dump, per:

cri-o/cmd/crio/main.go

Lines 40 to 48 in c4afa28

func writeCrioGoroutineStacks() {

path := filepath.Join("/tmp", fmt.Sprintf(

"crio-goroutine-stacks-%s.log",

strings.ReplaceAll(time.Now().Format(time.RFC3339), ":", ""),

))

if err := utils.WriteGoroutineStacksToFile(path); err != nil {

logrus.Warnf("Failed to write goroutine stacks: %s", err)

}

}

Users should guarantee that they have access to /tmp and that there is enough disk space available there.

We could make the goroutine dump a first-class citizen and include an endpoint for it, for example, supporting both the /dump/heap and /dump/threads. Both of these could support their location and pattern configuring via the crio.conf.

Few thoughts here following... Just thinking out loud.

I personally don't like this feature, as it can be abused as a vector for a potential DoS attack - calling this endpoint repeatedly could lead to disk space exhaustion, not to mention that there is a pause every time a heap dump has to be taken, thus there is a real impact on the application too. There is no authentication (basic or TLS certificates based) for these sorts of endpoints.

You could also then say - "oh, but it's only exposed via localhost!". Well, then, how useful this new feature is, given that the user can enable the option to expose pprof endpoints via the Unix socket, which is far more secure in contrast.

The ability to dump gorotines through sending SIGUSR1 to CRI-O is somewhat guarded through the notion of having to have proper permissions to send a signal to a different process, so the surface area open for abuse is a bit smaller.

Appreciate your insights on this. Using /tmp does have its conveniences, and the goroutine dump feature, as it stands, has some considerations, especially regarding potential abuse as a vector for DoS attacks. Your points about disk space exhaustion and the impact on application performance are well taken.

I wonder if there's an opportunity to strike a balance—perhaps by providing more secure alternatives for configuring the dump location and pattern in crio.conf, and simultaneously addressing the potential risks associated with frequent calls to the endpoint. Additionally, considering authentication measures for these endpoints could enhance security.

It might be beneficial to discuss this with @mrunalp, the original contributor to this feature. Understanding the initial vision and intentions behind the pull request would provide valuable context, and we can collectively explore the best approach that aligns with both security and usability considerations.

@mrunalp, could we dive into a discussion about the considerations around the goroutine dump feature and its potential impact on security and performance?

generally, if a malicious user has access to the crio socket, they can do harm in a lot of different ways. admins should ensure only trusted users can access nodes. For instance, a user could DOS CRI-O by having systemd shut it off. Since this and the goroutine profiles are both debugging options, I think it's safe to guess a user who has access can be trusted to not DOS the node's memory

let's mirror the other USR/HUP signal handling for now and have cri-o write to disk in an opinionated structure. we can evaluate other methods in the future and rewrite all of them

@hernandanielg, we've had some internal conversations about how to move forward with this feature.

There is, of course, still an opportunity to complete this work here, so please be sure to bear with us a little bit. 😄

That said, I would like to bring everyone's attention to the following discussion:

Make SIGUSR2 trigger a heap dump instead of triggering a garbage colletion (GC) #7615

Ok looking forward ;)

hey any news about this?

@hernandanielg, sorry for the delay!

A decision on how to proceed has stalled, so it seems. Apologies!

haircommander · 2023-12-04T20:35:55Z

@hernandanielg are you still working on this? if not, @shipra101 could take it over

hernandanielg · 2023-12-04T20:38:16Z

Yes, I'm sorry for the delay, please gimme until tomorrow to update this PR

Signed-off-by: Hernan Garcia <[email protected]>

codecov · 2023-12-06T08:14:07Z

Codecov Report

Merging #7328 (4a618b6) into main (1e93fb4) will decrease coverage by 0.02%.
Report is 330 commits behind head on main.
The diff coverage is 45.45%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7328      +/-   ##
==========================================
- Coverage   48.68%   48.67%   -0.02%     
==========================================
  Files         145      145              
  Lines       15918    15940      +22     
==========================================
+ Hits         7749     7758       +9     
- Misses       7235     7246      +11     
- Partials      934      936       +2

openshift-merge-robot · 2024-02-11T15:13:10Z

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

github-actions · 2024-04-01T00:03:12Z

A friendly reminder that this PR had no activity for 30 days.

hernandanielg requested a review from mrunalp as a code owner September 22, 2023 12:21

openshift-ci bot requested review from klihub and sohankunkerkar September 22, 2023 12:22

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 22, 2023

hernandanielg force-pushed the add-enable-heap-dump branch from cc7fb40 to b7de844 Compare September 22, 2023 12:25

openshift-ci bot added dco-signoff: yes Indicates the PR's author has DCO signed all their commits. and removed dco-signoff: no Indicates the PR's author has not DCO signed all their commits. labels Sep 22, 2023

hernandanielg marked this pull request as draft September 22, 2023 12:30

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 22, 2023

hernandanielg force-pushed the add-enable-heap-dump branch from b7de844 to a3895ca Compare October 24, 2023 15:35

kwilczynski reviewed Nov 6, 2023

View reviewed changes

hernandanielg force-pushed the add-enable-heap-dump branch from a3895ca to 0b9b296 Compare December 5, 2023 03:08

added enable-heap-dump flag

4a618b6

Signed-off-by: Hernan Garcia <[email protected]>

hernandanielg force-pushed the add-enable-heap-dump branch from 0b9b296 to 4a618b6 Compare December 5, 2023 03:56

hernandanielg requested a review from kwilczynski December 5, 2023 03:56

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 11, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 1, 2024

hernandanielg closed this Apr 19, 2024

hernandanielg mentioned this pull request May 7, 2024

Add an endpoint for heap dumps #7307

Closed

	func writeCrioGoroutineStacks() {
	path := filepath.Join("/tmp", fmt.Sprintf(
	"crio-goroutine-stacks-%s.log",
	strings.ReplaceAll(time.Now().Format(time.RFC3339), ":", ""),
	))
	if err := utils.WriteGoroutineStacksToFile(path); err != nil {
	logrus.Warnf("Failed to write goroutine stacks: %s", err)
	}
	}

enable heap dumps #7328

enable heap dumps #7328

Uh oh!

Conversation

hernandanielg commented Sep 22, 2023

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

openshift-ci bot commented Sep 22, 2023

Uh oh!

openshift-ci bot commented Sep 22, 2023

Uh oh!

openshift-ci bot commented Sep 22, 2023

Uh oh!

sohankunkerkar commented Sep 25, 2023

Uh oh!

sohankunkerkar commented Oct 4, 2023

Uh oh!

hernandanielg commented Oct 4, 2023

Uh oh!

hernandanielg commented Oct 24, 2023

Uh oh!

sohankunkerkar commented Oct 26, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haircommander commented Dec 4, 2023

Uh oh!

hernandanielg commented Dec 4, 2023

Uh oh!

codecov bot commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

openshift-merge-robot commented Feb 11, 2024

Uh oh!

github-actions bot commented Apr 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Dec 6, 2023 •

edited

Loading