Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

mikesmithgh
Copy link
Collaborator

@mikesmithgh mikesmithgh commented Feb 13, 2022

I would like to add leader election. Due to node autoscaler and other variables in the cluster, the metacontroller pod can get rolled/shifted to other nodes. Typically, this does not cause any problems and metacontroller comes up fast. However, we occasionally have issues with inconsistency between environment and bad nodes which may cause metacontroller to be stuck in a pending state for a while. This would allow for more coverage and reduce the chance of downtime in this scenario.

@codecov
Copy link

codecov bot commented Feb 13, 2022

Codecov Report

Merging #442 (2d9198f) into master (3b690b2) will decrease coverage by 0.05%.
The diff coverage is 37.50%.

❗ Current head 2d9198f differs from pull request most recent head 95b3337. Consider uploading reports for the commit 95b3337 to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #442      +/-   ##
==========================================
- Coverage   49.33%   49.27%   -0.06%     
==========================================
  Files          56       56              
  Lines        4966     4980      +14     
==========================================
+ Hits         2450     2454       +4     
- Misses       2265     2275      +10     
  Partials      251      251              
Flag Coverage Ξ”
integration 43.22% <100.00%> (+0.15%) ⬆️
unit 28.31% <0.00%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Ξ”
main.go 0.00% <0.00%> (ΓΈ)
pkg/server/server.go 58.57% <100.00%> (+2.51%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Ξ” = absolute <relative> (impact), ΓΈ = not affected, ? = missing data
Powered by Codecov. Last update 3b690b2...95b3337. Read the comment docs.

@mikesmithgh
Copy link
Collaborator Author

HI @grzesuav and @AmitKumarDas, could you please review when you have free time? I left this in draft because I have TODOs around naming and will update after feedback.

@grzesuav
Copy link
Contributor

hi @mjsmith1028 , it would be good to have consistency with https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/leaderelection/leader_election.go.

About cmd line options, I would make them the same as config options (i.e. leaderElectionEnabled, etc). I didn't went trough leader election in ctrl-runtime so cannot give more precise advice at the moment. I am aware that multiple resources can be used as leader key object, please make sure to use newest ones (I remember that ctrl-runtime is going away from configmaps to another resources)

Thanks for PR!

@AmitKumarDas
Copy link
Contributor

AmitKumarDas commented Feb 16, 2022

@mjsmith1028 Did you run some tests to verify if nothing breaks. Is it feasible for you to add some tests / examples that can verify this behaviour?
@grzesuav Is metacontroller now making use of controller-runtime such that just exposing the latter's arguments enable metacontroller to achieve what controller-runtime is capable of?

@mikesmithgh
Copy link
Collaborator Author

hi @mjsmith1028 , it would be good to have consistency with https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/leaderelection/leader_election.go.

About cmd line options, I would make them the same as config options (i.e. leaderElectionEnabled, etc). I didn't went trough leader election in ctrl-runtime so cannot give more precise advice at the moment. I am aware that multiple resources can be used as leader key object, please make sure to use newest ones (I remember that ctrl-runtime is going away from configmaps to another resources)

Thanks for PR!

@grzesuav Thanks! I updated to expose all the leader election options and defaulted to the preferred resource lock leases. I'll add some documentation around this. I am wondering if there is any concern about the growing number of arguments passed to metacontroller or if you are fine with that approach?

@mikesmithgh
Copy link
Collaborator Author

@mjsmith1028 Did you run some tests to verify if nothing breaks. Is it feasible for you to add some tests / examples that can verify this behaviour? @grzesuav Is metacontroller now making use of controller-runtime such that just exposing the latter's arguments enable metacontroller to achieve what controller-runtime is capable of?

@AmitKumarDas I only ran some small manual testing around this to verify a leader was elected and the other replica waits until the leader is in a bad state before becoming the new leader. I can definitely add an example. I'm not 100% sure on the approach I should take for testing but I'll think on it. Do you have any ideas?

@mikesmithgh mikesmithgh force-pushed the leader-election branch 2 times, most recently from ea80314 to b2019f0 Compare March 4, 2022 04:42
@mikesmithgh mikesmithgh marked this pull request as ready for review March 4, 2022 04:56
@mikesmithgh mikesmithgh requested a review from AmitKumarDas March 4, 2022 04:56
@mikesmithgh
Copy link
Collaborator Author

Hi @AmitKumarDas and @grzesuav, I cleaned up this PR and it is ready to review. Please let me know if you have anymore feedback.

@mikesmithgh mikesmithgh force-pushed the leader-election branch 2 times, most recently from 2d9198f to 481f378 Compare March 7, 2022 19:41
@mikesmithgh
Copy link
Collaborator Author

FYI I rebased to clean up the commit history.

Signed-off-by: Mike Smith <[email protected]>
Copy link
Contributor

@grzesuav grzesuav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good and clear, thanks for this PR !

@grzesuav grzesuav merged commit 29563b2 into metacontroller:master Mar 8, 2022
@grzesuav
Copy link
Contributor

grzesuav commented Mar 8, 2022

πŸŽ‰ This PR is included in version 2.3.0 πŸŽ‰

The release is available on GitHub release

Your semantic-release bot πŸ“¦πŸš€

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants