Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Do not interrupt querying readiness probes for plugins #6975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chrisohaver merged 4 commits into from
Apr 8, 2025
Merged

Do not interrupt querying readiness probes for plugins #6975

chrisohaver merged 4 commits into from
Apr 8, 2025

Conversation

@glebkin
Copy link
Contributor

@glebkin glebkin commented Nov 15, 2024 •
edited
Loading

1. Why is this pull request needed and what does it do?

Kubernetes constantly polls the readiness of the applications running in cluster. Getting the OK status does not guarantee that the application will continue to be ready to serve traffic. For example, our DNS server can use an external application as a source for DNS records. At some point, it may stop responding, we might want to signal this to k8s and stop receiving traffic until the connection is restored.

2. Which issues (if any) are related?

No issues

3. Which documentation changes (if any) need to be made?

I've already updated documentation for ready plugin

4. Does this introduce a backward incompatible change or deprecation?

It's backwards compatible as I introduced new functionality via additional configuration param.

@codecov
Copy link

codecov bot commented Nov 21, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 77.27273% with 10 lines in your changes missing coverage. Please review.

Project coverage is 56.59%. Comparing base (93c57b6) to head (5c77bb3).
Report is 1411 commits behind head on master.

Files with missing lines Patch % Lines
plugin/ready/setup.go 69.69% 9 Missing and 1 partial ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #6975      +/-   ##
==========================================
+ Coverage   55.70%   56.59%   +0.89%     
==========================================
  Files         224      270      +46     
  Lines       10016    17408    +7392     
==========================================
+ Hits         5579     9852    +4273     
- Misses       3978     6965    +2987     
- Partials      459      591     +132

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@glebkin
Copy link
Contributor Author

glebkin commented Jan 9, 2025

Guys, would you please kindly take a look into this PR?

@chrisohaver
Copy link
Member

chrisohaver commented Jan 9, 2025

I didn't find any plugins in CoreDNS implementing ready.Readiness interface.

The kubernetes plugin does. This is a very significant implementation that we want to be extra careful not to introduce changes in behavior that could disrupt existing processes.

@chrisohaver
Copy link
Member

chrisohaver commented Jan 9, 2025

I think this may change the existing behavior of Kubernetes readiness checks. It depends on the behavior of the HasSynced implementations in the k8s client API. This change might cause a pod to start reporting unready if HasSynced ever becomes false after it is true after the initial synchronization following connection. This may not be a problematic change, but we still need to be extra careful if introducing any change in behavior.

To be safe, we could bake the old behavior into the kubernetes readiness implementation (e.g. once HasSynced is true, then always return true for subsequent calls without checking HasSynced again).

@glebkin
Copy link
Contributor Author

glebkin commented Jan 9, 2025

We could also introduce a new flag in the ready plugin to make it backward compatible. If this is true, we will not remove the plugin from the list if its Ready implementation returns true.

@chrisohaver
Copy link
Member

chrisohaver commented Jan 9, 2025

We could also introduce a new flag in the ready plugin to make it backward compatible. If this is true, we will not remove the plugin from the list if its Ready implementation returns true.

yes - as long as the default behavior is the old behavior, this would work as well - but it does add one more mostly superfluous option to the configuration.

@Shmillerov
Copy link
Contributor

Shmillerov commented Apr 7, 2025

Hello!

I think it would be right to make a ready plugin with regular checks.

From k8s docs: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes

Readiness probes runs on the container during its whole lifecycle.

I think it's not very good that CoreDNS ready plugin can't change the status at runtime.

@chrisohaver, @glebkin tell me what is needed to get things moving?

About kubernetes plugin with HasSynced(). I looked at the docs and impl and it looks like this parameter is set to true once and can't be changed anymore.

https://pkg.go.dev/k8s.io/client-go/tools/cache#SharedInformer

// HasSynced returns true if the shared informer's store has been
// informed by at least one full LIST of the authoritative state
// of the informer's object collection.  This is unrelated to "resync".
//
// Note that this doesn't tell you if an individual handler is synced!!
// For that, please call HasSynced on the handle returned by
// AddEventHandler.
HasSynced() [bool](https://pkg.go.dev/builtin#bool)

https://pkg.go.dev/k8s.io/client-go/tools/cache#FIFO.HasSynced

HasSynced returns true if an Add/Update/Delete/AddIfNotPresent are called first, or the first batch of items inserted by Replace() has been popped.

@chrisohaver
Copy link
Member

chrisohaver commented Apr 7, 2025

I’d prefer this be enabled by flag/option. Else we should defer it to the next minor release. How about monitor (until-ready | continuously)? I’m open to other name ideas.

@glebkin
Copy link
Contributor Author

glebkin commented Apr 8, 2025

I’d prefer this be enabled by flag/option. Else we should defer it to the next minor release. How about monitor (until-ready | continuously)? I’m open to other name ideas.

I've updated PR, please take a look

@glebkin
Update ready docs
903528b 
Signed-off-by: Gleb Kogtev <[email protected]>
chrisohaver
chrisohaver reviewed Apr 8, 2025
View reviewed changes
@glebkin
Update ready docs
5c77bb3 
Signed-off-by: Gleb Kogtev <[email protected]>
chrisohaver
chrisohaver reviewed Apr 8, 2025
View reviewed changes
@chrisohaver chrisohaver merged commit 52b3172 into coredns:master Apr 8, 2025
13 checks passed
@BrewTestBot BrewTestBot mentioned this pull request Jun 6, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrisohaver chrisohaver chrisohaver approved these changes

@bradbeam bradbeam Awaiting requested review from bradbeam bradbeam is a code owner

@dilyevsky dilyevsky Awaiting requested review from dilyevsky dilyevsky is a code owner

@jameshartig jameshartig Awaiting requested review from jameshartig jameshartig is a code owner

@greenpau greenpau Awaiting requested review from greenpau greenpau is a code owner

@Isolus Isolus Awaiting requested review from Isolus Isolus is a code owner

@johnbelamaric johnbelamaric Awaiting requested review from johnbelamaric johnbelamaric is a code owner

@miekg miekg Awaiting requested review from miekg miekg is a code owner

@pmoroney pmoroney Awaiting requested review from pmoroney pmoroney is a code owner

@rajansandeep rajansandeep Awaiting requested review from rajansandeep rajansandeep is a code owner

@stp-ip stp-ip Awaiting requested review from stp-ip stp-ip is a code owner

@SuperQ SuperQ Awaiting requested review from SuperQ SuperQ is a code owner

@yongtang yongtang Awaiting requested review from yongtang yongtang is a code owner

@Tantalor93 Tantalor93 Awaiting requested review from Tantalor93 Tantalor93 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@glebkin @chrisohaver @Shmillerov