Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Reduce offline volume expansion flake #94144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Sep 10, 2020
Merged

Reduce offline volume expansion flake #94144

merged 2 commits into from
Sep 10, 2020

Conversation

gnufied
Copy link
Member

@gnufied gnufied commented Aug 20, 2020 •
edited
Loading

This PR makes two fixes:

  • A volume can be expanded again if we still have stale PV in informer. In worst case this causes a flake because volume expansion will be retried.
  • It also increases time we wait for 2nd pod to startup because that time includes time for volume to detach from previous node and attach to new one.

/sig storage

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Aug 20, 2020
@k8s-ci-robot k8s-ci-robot requested review from msau42 and verult August 20, 2020 20:27
@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 20, 2020
@gnufied
Copy link
Member Author

gnufied commented Aug 20, 2020

/kind bug
/release-note none

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Aug 20, 2020
@gnufied
Copy link
Member Author

gnufied commented Aug 20, 2020

/release-note-none

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 20, 2020
@gnufied
Copy link
Member Author

gnufied commented Aug 20, 2020

/retest

@gnufied
Copy link
Member Author

gnufied commented Aug 25, 2020

/assign @msau42

@gnufied gnufied changed the title Read PV object from apiserver to prevent flake Reduce offline volume expansion flake Sep 10, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gnufied

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added area/test sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Sep 10, 2020
jsafrane
jsafrane reviewed Sep 10, 2020
View reviewed changes
pkg/controller/volume/expand/expand_controller.go Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you consider adding a PV to expc.pvLister after the PV is saved with the new size? Would that save us from an API call?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did. it is kind of complicated to do that because expansion call is handled by operation_generator(and we don't have informer there) and it will require some larger re-factoring to do that correctly. I think overall since this call is only made after it has been determined expansion is needed and we don't expect people to be resizing volumes that often, it is I think acceptable tradeoff.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack

@gnufied gnufied force-pushed the prevent-stale-pv-read branch from 2655491 to 4614817 Compare September 10, 2020 14:03
@jsafrane
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 10, 2020
@jsafrane
Copy link
Member

/milestone 1.20

@k8s-ci-robot
Copy link
Contributor

@jsafrane: The provided milestone is not valid for this repository. Milestones in this repository: [next-candidate, v1.16, v1.17, v1.18, v1.19, v1.20, v1.21, v1.22]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jsafrane
Copy link
Member

/milestone v1.20

@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Sep 10, 2020
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit c7d6f79 into kubernetes:master Sep 10, 2020
This was referenced Sep 15, 2021
Closed
Merged
@carlory carlory mentioned this pull request Jul 18, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsafrane jsafrane jsafrane left review comments

@msau42 msau42 Awaiting requested review from msau42

@verult verult Awaiting requested review from verult

Assignees

@jsafrane jsafrane

@msau42 msau42

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v1.20
Development

Successfully merging this pull request may close these issues.
5 participants
@gnufied @k8s-ci-robot @jsafrane @fejta-bot @msau42