move filesystem resize code to kubernetes/mount-utils and add need re… #99223

AndyXiangLi · 2021-02-19T02:30:05Z

What type of PR is this?

/kind cleanup
/kind feature

What this PR does / why we need it:

Move filesystem resize code to kubernetes/mount-utils so that it can be shared among with CSI drivers.

Add NeedResize function in ResizeFS

Which issue(s) this PR fixes:

Fixes #94929

Still need effort on the plugin side

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

add NeedResize function to kubernetes/mount-utils, user can call this function to determine if fs need to be resized

k8s-ci-robot · 2021-02-19T02:30:07Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: [email protected]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-02-19T02:30:12Z

Welcome @AndyXiangLi!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-02-19T02:30:13Z

Hi @AndyXiangLi. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jingxu97 · 2021-02-20T00:45:14Z

/approve

jingxu97 · 2021-02-20T00:45:27Z

/assign @gnufied

gnufied · 2021-02-22T17:39:55Z

/ok-to-test

gnufied · 2021-02-22T17:41:07Z

/triage accepted
/priority important-soon
/kind cleanup

jsafrane · 2021-02-24T10:47:57Z

staging/src/k8s.io/mount-utils/resizefs_linux_test.go

+			blockCount:  0,
+			cmdOutput:   cmdOutputNoDataExt4,
+			expectError: true,
+			fsType:      "ext4",


nit: please add a test with unknown FS

gnufied · 2021-02-24T17:27:07Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

+			continue
+		}
+		key, value := strings.TrimSpace(tokens[0]), strings.TrimSpace(tokens[1])
+		if key == "Block count" {


Can we downcase before comparing them?

gnufied · 2021-02-24T17:37:02Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

+	lines := strings.Split(string(output), "\n")
+	var blockSize, blockCount uint64
+
+	for _, line := range lines {


cc @sandeen for xfs_info parsing.

Why not just use statfs to get the size of /any/ filesystem, instead of hard-coding and scraping individual filesystem utilities? statfs is the generic interface to query the kernel about the size of a mounted filesystem.

See statfs(2) and the -f option in statfs(1)

Hm, well, this doesn't count the log blocks I guess. Not sure what your threshold for "should try to grow" is.

Or even just:

df -B1 --output=size /boot

1B-blocks
1063256064

and again this ignores space consumed by journals or other internal fs housekeeping, but it may be better to use standard APIs like statfs(2) (which is how df gets its information)

@sandeen, I may miss something here, but IMO statfs shows size of the filesystem available for actual files (i.e. without journal and other FS metadata), while xfs_info shows the actual size occupied by the filesystem on a device, incl. journal, inodes, and whatnot.

We need to reliably check if we should call resize2fs / xfs_growfs, i.e. there is some extra space on the device that the filesystem does not know about.

$ df --output=size /boot 1K-blocks 999320 $ dumpe2fs /dev/nvme0n1p2 Block count: 262144 Block size: 4096

999320 * 1024 = 1023303680
262144 * 4096 = 1073741824

That's quite a difference.

just try a growfs after each initial mount, which is going to be quick and harmless if the filesystem is already maximally sized.

How safe is this for xfs and ext3/ext4? Since we will do it with every mount() and containers can move around the cluster rather quickly, we need it to be fast enough (say same as mount()?) and reliable (won't ever harm the FS, even when it gets SIGINT).

What is your typical image size, roughly?

It starts with 1 GB and goes to hundreds of GB. But I am not sure it's relevant. If user extends the volume by 1% (which can be say 1GB), we'd like the volume to be be expanded.

Let's chat on IRC when you get a chance, higher bandwidth than github comments might help.

summary of conversation on IRC, here are the options:

<sandeen> 1) use df, and accept a small % error <sandeen> 2) scrape filesystem utilities and hope they don't change <sandeen> 3) always call resize, which would be a no-op if already maximally sized <sandeen> 4) use filesystem-specific but more well-defined APIs where they exist <sandeen> ... <sandeen> 5) wait for a common kernel solution :(

We'd prefer 4) if available and for xfs we were able to find xfs_io -c statfs <mount> , which is much easier to parse. In addition, it uses ioctl to get the geometry. @AndyXiangLi can you try ioctl with some interface to make unit testing possible?

There is no ioctl for ext3/4, so 2) is IMO the best option and luckily dumpe2fs is not that horrible to parse.

Correction: is go & ioctl portable to ARM / PPC / s390x and other weird architectures that Kubernetes supports? If not, do not hesitate to fall back to parsing xfs_io -c statfs.

If you parse the xfs_io output, the field you want is "geom.datablocks" to get the actual, complete number of all blocks on the filesytem's data device. Note that this is only available when running the xfs_io "statfs" command on an xfs filesystem, non-xfs filesystems will only return the basic statfs(2) information, same as df.

AndyXiangLi · 2021-02-24T20:19:37Z

/test pull-kubernetes-e2e-kind

sandeen

After talking w/ @gnufied I'm realizing that I may not understand the use case / workflow here.
I /can/ help with any questions about xfs_info parsing, but it concerns me that this may not be the optimal approach here.

not used to github review flow so not sure if I created something blocking here, I'll just, er, submit it now.

sandeen · 2021-02-24T20:02:01Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

+	lines := strings.Split(string(output), "\n")
+	var blockSize, blockCount uint64
+
+	for _, line := range lines {


Why not just use statfs to get the size of /any/ filesystem, instead of hard-coding and scraping individual filesystem utilities? statfs is the generic interface to query the kernel about the size of a mounted filesystem.

See statfs(2) and the -f option in statfs(1)

sandeen · 2021-02-24T20:06:14Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

+	lines := strings.Split(string(output), "\n")
+	var blockSize, blockCount uint64
+
+	for _, line := range lines {


Hm, well, this doesn't count the log blocks I guess. Not sure what your threshold for "should try to grow" is.

sandeen · 2021-02-24T20:32:56Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

+	lines := strings.Split(string(output), "\n")
+	var blockSize, blockCount uint64
+
+	for _, line := range lines {


Or even just:

df -B1 --output=size /boot

1B-blocks
1063256064

and again this ignores space consumed by journals or other internal fs housekeeping, but it may be better to use standard APIs like statfs(2) (which is how df gets its information)

AndyXiangLi · 2021-03-12T18:10:01Z

/test pull-kubernetes-e2e-kind-ipv6

AndyXiangLi · 2021-03-12T20:48:49Z

@sandeen @jsafrane Sorry for the delay, was pulled to another priority last week. Just updated the command to xfs_io -c statfs <mount_path> and re-parsing the out put. Thank you for looking into this!

AndyXiangLi · 2021-04-08T16:54:57Z

@jsafrane Any updates?

jsafrane

LGTM, just one small nit.

jsafrane · 2021-04-12T18:52:21Z

staging/src/k8s.io/mount-utils/resizefs_linux.go

nit: why empty comment?

Thank you for review! This is updated

…size function

AndyXiangLi · 2021-04-12T23:16:49Z

/test pull-kubernetes-e2e-kind

jsafrane · 2021-04-13T18:05:13Z

/lgtm

AndyXiangLi · 2021-04-13T19:37:31Z

/test pull-kubernetes-integration

AndyXiangLi · 2021-04-13T20:18:13Z

/test pull-kubernetes-e2e-kind-ipv6

…#99223-upstream-release-1.21 Automated cherry pick of #99223: move filesystem resize code to kubernetes/mount-utils and add

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 19, 2021

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Feb 19, 2021

k8s-ci-robot requested review from dims and jingxu97 February 19, 2021 02:30

k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 19, 2021

AndyXiangLi force-pushed the need-resize branch from 75130ae to ac62aca Compare February 19, 2021 02:31

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 19, 2021

AndyXiangLi mentioned this pull request Feb 19, 2021

Resize filesystem when restore a snapshot to larger size volume kubernetes-sigs/aws-ebs-csi-driver#753

Merged

k8s-ci-robot assigned gnufied Feb 20, 2021

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 22, 2021

jsafrane reviewed Feb 24, 2021

View reviewed changes

gnufied reviewed Feb 24, 2021

View reviewed changes

AndyXiangLi force-pushed the need-resize branch 2 times, most recently from 1f09c4c to 89d12eb Compare February 24, 2021 18:41

sandeen reviewed Feb 25, 2021

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 1, 2021

AndyXiangLi force-pushed the need-resize branch 2 times, most recently from 4a1ab84 to 42cee7e Compare March 12, 2021 02:20

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 12, 2021

AndyXiangLi force-pushed the need-resize branch from 42cee7e to d0c2b8f Compare March 12, 2021 17:24

jsafrane reviewed Apr 12, 2021

View reviewed changes

move filesystem resize code to kubernetes/mount-utils and add need re…

03106a1

…size function

AndyXiangLi force-pushed the need-resize branch from d0c2b8f to 03106a1 Compare April 12, 2021 21:37

k8s-ci-robot assigned jsafrane Apr 13, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 13, 2021

k8s-ci-robot merged commit 4f6c595 into kubernetes:master Apr 13, 2021

k8s-ci-robot added this to the v1.22 milestone Apr 13, 2021

AndyXiangLi mentioned this pull request Apr 19, 2021

Automated cherry pick of #99223: move filesystem resize code to kubernetes/mount-utils and add #101253

Merged

k8s-ci-robot added a commit that referenced this pull request May 8, 2021

Merge pull request #101253 from AndyXiangLi/automated-cherry-pick-of-…

5e853bf

…#99223-upstream-release-1.21 Automated cherry pick of #99223: move filesystem resize code to kubernetes/mount-utils and add

github-actions bot mentioned this pull request May 11, 2021

Week Ending May 9, 2021 dev-obs/actus#370

Open

jsafrane mentioned this pull request Feb 1, 2022

Resize filesystems automatically when the block device allows bigger size #99042

Closed

move filesystem resize code to kubernetes/mount-utils and add need re… #99223

move filesystem resize code to kubernetes/mount-utils and add need re… #99223

Uh oh!

Conversation

AndyXiangLi commented Feb 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Feb 19, 2021

Uh oh!

k8s-ci-robot commented Feb 19, 2021

Uh oh!

k8s-ci-robot commented Feb 19, 2021

Uh oh!

jingxu97 commented Feb 20, 2021

Uh oh!

jingxu97 commented Feb 20, 2021

Uh oh!

gnufied commented Feb 22, 2021

Uh oh!

gnufied commented Feb 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

df -B1 --output=size /boot

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndyXiangLi commented Feb 24, 2021

Uh oh!

sandeen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

df -B1 --output=size /boot

Uh oh!

AndyXiangLi commented Mar 12, 2021

Uh oh!

AndyXiangLi commented Mar 12, 2021

Uh oh!

AndyXiangLi commented Apr 8, 2021

Uh oh!

jsafrane left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

AndyXiangLi commented Feb 19, 2021 •

edited

Loading