Codecov Report

Merging #376 (a9968b0) into master (81ac1ad) will increase coverage by 3.20%.
The diff coverage is 93.66%.

@@            Coverage Diff             @@
##           master     #376      +/-   ##
==========================================
+ Coverage   46.52%   49.72%   +3.20%     
==========================================
  Files          49       55       +6     
  Lines        3196     3306     +110     
==========================================
+ Hits         1487     1644     +157     
+ Misses       1461     1420      -41     
+ Partials      248      242       -6     

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 81ac1ad...a9968b0. Read the comment docs.

hi @mjsmith1028 , will take a look tommorow, thanks for the PR

Hi @grzesuav, I still haven't had time to fully dig into this. But, I played around a little today and I think I see what is happening at least in one case.

1. https://github.com/metacontroller/metacontroller/blob/master/pkg/controller/composite/controller.go#L543 - removes the finalizer
2. k8s deletes the object
3. https://github.com/metacontroller/metacontroller/blob/master/pkg/controller/composite/controller.go#L598 - attempts to update the object but it no longer exists

The reason I started seeing this is I bumped up the go-client-qps/burst to 150/300 and num workers to 50 so we are processing workloads very fast. In this case, the object is deleted so fast that it can trigger this scenario occasionally.

I added a time.Sleep(1 * time.Minute) before updateParentStatus to reproduce this.

2021-10-19T00:56:39.728Z	ERROR	failed to sync IndexedJob 'default/print-index': can't update status for IndexedJob default/print-index: indexedjobs.ctl.enisoc.com "print-index" not found

metacontroller/pkg/controller/composite.(*parentController).worker
	/go/src/metacontroller/pkg/controller/composite/controller.go:287
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
metacontroller/pkg/controller/composite.(*parentController).Start.func1.1
	/go/src/metacontroller/pkg/controller/composite/controller.go:263

I'll try to get more concrete details on this. But, at least in my local branch I have added checks for 404 on deletes, 409 on updates, and 403 on creates. I'll update the PR with some more details later this week.

@mjsmith1028 yes, it can be related to fact that client-go uses cache state of the cluster to not kill api-server, so it can see not up-to-date state of the cluster. discovery-interval is set by default to 30seconds

to be more exact, informers (which are triggering reconcile) uses cache version of the k8s-client, but operations in controller interacts with real-time cluster.

I have just doubts in those two places if we can safely swallow the error, could you comment ?

Hi @grzesuav , I added some comments on why I think it should be safe.

One thing to note is that my metacontroller setup has 50 workers, 150 client-go qps and 300 client-go-burst. So, it is processing sync requests pretty fast and the cluster is pretty active with other work outside of my controller which may put some stress on K8s control plane causing occasional slowness in K8s API responses. I think that is why these edge cases showed up.

Please let me know if the logic seems correct in my comments. Thanks!

🎉 This PR is included in version 2.0.15 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀