fix race in aggregated discovery controller #115302

alexzielenski · 2023-01-24T22:32:05Z

Discovered by flaking TestRemoveAPIServer.

Caused by following sequence:

Add APIService to map
Begin Async Fetch (takes long time in flaking case)
Remove APIService from map
Finish Async Fetch & stores apiservice back in map to update lastReconciled time

fixes by removing lastReconciled (only used for testing) so it would not need to be updated.
and switching tests to just waiting until dirty queue is empty

had to put sleeps in the TestRemoveAPIService to be able to trigger the bug as well as after fetchFreshDiscoveryForService to simulate long-running request, as stress could not repro on my system

What type of PR is this?

/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

flaky test initially discovered in #115230 (comment)

but its also a bug outside of tests

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

alexzielenski · 2023-01-24T22:32:30Z

/sig api-machinery
/cc @Jefftree @apelisse

alexzielenski · 2023-01-24T22:38:42Z

staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_discovery.go

this was the problematic line of code.

if apiservice was removed (for instance during long-running fetchFreshDiscoveryForService) then this would then add it back erroneously

Is lastReconciled ever set now?

It has been removed. It was only used for tests and replaced by AllServicesSynced

apelisse · 2023-01-25T00:35:40Z

I think that's good, thanks!

apelisse · 2023-01-25T01:09:53Z

/lgtm

k8s-ci-robot · 2023-01-25T01:09:59Z

LGTM label has been added.

Git tree hash: aaf23b9733607a1ea9a2d1394dcc19a0ce4d0541

alexzielenski · 2023-01-25T01:11:33Z

/assign @lavalamp
please look if you have time

staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_discovery.go

lavalamp · 2023-01-25T01:14:16Z

staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_discovery.go

if it is for testing can you make it not exported and/or move it to live in the _test.go file?

That would require making the tests whitebox. Wasnt sure of the correct side of the tradeoff to take. Ill change it over then

that's unfortunate but I don't think you want external callers of this as it is.

I'd rather have the test use an internal method than a function exported for the sake of an external test?

correct. im moving it over

I've removed AllServicesSynced and replaced with internal test function

Caused by following sequence: 1. Add APIService to map 2. Begin Async Fetch 3. Remove APIService from map 4. Finish Async Fetch & stores apiservice back in map to update lastReconciled time fixes by removing lastReconciled (only used for testing) and switching tests to just waiting until dirty queue is empty

alexzielenski · 2023-01-25T15:07:07Z

/retest

PTAL again

lavalamp · 2023-01-25T18:11:01Z

staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_discovery_test.go

 )

+func newDiscoveryManager(rm discoveryendpoint.ResourceManager) *discoveryManager {
+	return NewDiscoveryManager(rm).(*discoveryManager)


Something to think about for a followup: why does this package need both an interface and an implementation struct? Why not just an exported implementation struct?

lavalamp · 2023-01-25T18:11:39Z

/lgtm
/approve
/retest

k8s-ci-robot · 2023-01-25T18:11:45Z

LGTM label has been added.

Git tree hash: 366bcfee710586f5919056696d681f1416554868

k8s-ci-robot · 2023-01-25T18:12:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexzielenski, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/kube-aggregator/OWNERS~~ [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-triage-robot · 2023-01-25T20:33:52Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

k8s-triage-robot · 2023-01-25T22:54:52Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

alexzielenski · 2023-01-25T23:08:52Z

/retest

k8s-triage-robot · 2023-01-26T01:20:53Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

k8s-triage-robot · 2023-01-26T04:29:53Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

alexzielenski · 2023-01-26T06:23:51Z

/retest

cici37 · 2023-01-26T17:38:45Z

/triage accepted

liggitt · 2023-02-14T22:17:52Z

Thanks for the fix ... the test is still flaking on release-1.26 (https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=TestRemoveAPIService)... can we backport the fix if only to stabilize CI?

…f-#115302-upstream-release-1.26 Cherry pick of Aggregated Discovery Patches: #115302 #115770 #115998 #115859

k8s-ci-robot requested review from apelisse and Jefftree January 24, 2023 22:32

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 24, 2023

k8s-ci-robot requested review from caesarxuchao and smarterclayton January 24, 2023 22:33

alexzielenski commented Jan 24, 2023

View reviewed changes

alexzielenski changed the title ~~fix race in aggregated discovery handler~~ fix race in aggregated discovery controller Jan 24, 2023

k8s-ci-robot assigned apelisse Jan 25, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 25, 2023

k8s-ci-robot assigned lavalamp Jan 25, 2023

lavalamp reviewed Jan 25, 2023

View reviewed changes

staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_discovery.go Outdated Show resolved Hide resolved

lavalamp reviewed Jan 25, 2023

View reviewed changes

alexzielenski force-pushed the apiserver/discovery/lastreconciled-race branch from 08cb6d3 to cff4d07 Compare January 25, 2023 01:33

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 25, 2023

k8s-ci-robot requested a review from lavalamp January 25, 2023 01:33

lavalamp reviewed Jan 25, 2023

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 25, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 25, 2023

k8s-ci-robot merged commit eb4e2a2 into kubernetes:master Jan 26, 2023

k8s-ci-robot added this to the v1.27 milestone Jan 26, 2023

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 26, 2023

alexzielenski mentioned this pull request Jan 27, 2023

Fix TestRemoveAPIService test in discovery aggregation #114442

Closed

alexzielenski mentioned this pull request Feb 15, 2023

Cherry pick of Aggregated Discovery Patches: #115302 #115770 #115998 #115859 #115805

Merged

gjkim42 mentioned this pull request Feb 27, 2023

Deflake tests in staging/src/k8s.io/kube-aggregator/pkg/apiserver #115859

Merged

k8s-ci-robot added a commit that referenced this pull request Mar 11, 2023

Merge pull request #115805 from alexzielenski/automated-cherry-pick-o…

44c0247

…f-#115302-upstream-release-1.26 Cherry pick of Aggregated Discovery Patches: #115302 #115770 #115998 #115859

pacoxu mentioned this pull request Nov 13, 2023

Failure cluster [46587915...] TestRemoveAPIService unit test flaking #114438

Closed

fix race in aggregated discovery controller #115302

fix race in aggregated discovery controller #115302

Uh oh!

Conversation

alexzielenski commented Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

alexzielenski commented Jan 24, 2023

Uh oh!

alexzielenski Jan 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexzielenski Jan 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apelisse commented Jan 25, 2023

Uh oh!

apelisse commented Jan 25, 2023

Uh oh!

k8s-ci-robot commented Jan 25, 2023

Uh oh!

alexzielenski commented Jan 25, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexzielenski commented Jan 25, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lavalamp commented Jan 25, 2023

Uh oh!

k8s-ci-robot commented Jan 25, 2023

Uh oh!

k8s-ci-robot commented Jan 25, 2023

Uh oh!

k8s-triage-robot commented Jan 25, 2023

Uh oh!

k8s-triage-robot commented Jan 25, 2023

Uh oh!

alexzielenski commented Jan 25, 2023

Uh oh!

k8s-triage-robot commented Jan 26, 2023

Uh oh!

k8s-triage-robot commented Jan 26, 2023

Uh oh!

alexzielenski commented Jan 26, 2023

Uh oh!

cici37 commented Jan 26, 2023

Uh oh!

liggitt commented Feb 14, 2023

Uh oh!

Uh oh!

alexzielenski commented Jan 24, 2023 •

edited

Loading

alexzielenski Jan 24, 2023 •

edited

Loading

alexzielenski Jan 25, 2023 •

edited

Loading