Is it possible to expand this comment explaining why is this required? It would be nice to detail the expected sequence of events that proves the tracker is needed when both operator and agent manage CIDs.

Updated the docs

I'm sorry but I'm not following the comment.

AFAIU we need the delay (and thus the marking and deletion-tracking) to avoid scenarios where the agent created the CID but it does not have updated the CEP with the CID yet. If the operator sees the new CID and no CEP actually using it, it might be too aggressive and delete the CID. This is supposed to happen more frequently in case of high pod churn (I guess because this increases the delay between the operator receiving the CID creation event and the operator receiving the CEP update event).
Is my understanding correct? Either way, I suggest to rephrase the comment to be more clear and descriptive.

It address two main concerns:

1. It facilitates CID reuse in scenarios with high pod churn. We are not deleting the CID and we are able to re-use it.
2. Ensure correct CID association in scenarios where the agent rapidly receives create, delete events.
   e.g. we have 1 CID used by one pod, and we churn this pod (agents quickly receives create, delete CID events). If the agent receives the delete CID, schedules the CEP deletion, then a new create event comes, sees that the CEP is already existing, the scheduled CEP event deletes the CEP.

@pippolo84, added more details in the comment. I think once we have the full implementation it will provide enough context.

I also consider the delayed deletion is useful in high pod churn scenarios as we do not have to delete and re-create the same CID over and over..

sure, but that's already implemented in the current GC.

Updated the docs.

@dlapcevic do you have any thoughts around the existing GC and the one proposed to be in operator managing CIDs?

I will try to sum it up and let me know if my understanding is correct so @joestringer and @dlapcevic doesn't have to read all of that conversation 😅

We are considering how GC behaves in two different cases:
(1) During the transition period from cilium-agent creating CIDs to operator creating CIDs when both operator and agent can create CIDs
(2) How GC behaves once operators is fully managing CIDs

Let's think about the current GC and how it would behave:
(1) if CID was ever used, it will be marked for deletion after 30 minutes of not being used with annotation and deleted after another 15 minutes - this already addresses your concern with high-churn pods.
If CID was never used by any CEP, it will mark it for deletion and delete it after 15 minutes.
Sounds reasonable both for identities created by the operator and the agent.
CIDs created by the agent and not yet used by any CEPs won't be deleted immediately.

(2) It will still work the same, but adding an annotation mark for deletion won't make much sense as the agent won't be able to remove it anymore.
So proposed switch from deletion mark annotation to DeletionTracker is a nice optimization in terms of API calls, but not necessary for it to work correctly. It should be fairly easy to implement it in the current GC though.

It could probably even be instantly enabled ( deletion mark annotation -> DeletionTracker) when switching from agent creating CIDs to operator creating CIDs during migration and the current GC would still behave correctly in both cases (1) and (2), but would not react to deletion mark annotation, but used fixed delay for waiting on CEP.

Of course interval/timeouts (30/15 minutes) can be already changed IIRC, I was just using defaults as examples.

So the question is, what would be the purpose of implementing a new GC, instead of implementing this improvement to the current GC, which seems significantly easier?

In my opinion, GC is part of CID management, and there is no reason for it to be separated from the CID controller. The CID controller anyway needs to know the desired state of CIDs, when CIDs are used and when not, and adding deletion to it is not a complex effort.

Beside that I see a few benefits:

1. Performance and scalability
   a) Smaller number of calls to kube-apiserver without adding and removing the mark. Remember: every update needs to be sent to all nodes.
   b) Processing all CIDs at one time (each GC run) instead of continuously, creates a spike of requests that can significantly affect KCP and cilium-operator's k8s client’s rate limiting.
2. Ease of use
   No need to configure GC.
3. Security
   Remove write permission for CIDs from cilium-agent.
   When cilium-agent is not managing CIDs, it shouldn’t update CIDs, so it won’t need write permission to CIDs.

We had issues related to these points.
We also had issues where cilium-operator was having connection instability with kube-apiserver so it was restarting every 15-20 minutes (or leader was changed), and in this case we ended up having no CIDs cleaned up and eventually hitting the 65k CID limit.

I think that this topic would be a lot more approachable with a more incremental approach. Use the current GC first, propose improvements to the current GC implementation and push on that independently. Then propose the PR that refactors/removes the current GC implementation to merge it into this CID controller. Each of those steps would be self-contained, provide value, and could be individually assessed for their benefits and drawbacks.

If we were looking at a PR that was titled "Improve cilium-operator Identity garbage collection" with the PR description containing the text in the post immediately above, then I think we would be in a much better position to critique the design and implementation, and debate how it affects the implementation.

Given how critical these operations are and how nuanced they can be, I don't think we're interested in having multiple implementations in the tree at the same time. Not unless there's something critical to the design that I'm missing which makes the garbage collection improvements inherently different.

EDIT: Let me adjust a bit on multiple GC: If we want to trial and incrementally roll out two implementations, then it could be useful to have a swappable implementation where you define which GC algorithm to use based on a flag. But for that, ideally we would just structure it so that the GC aspect fits into the broader code in a consistent way and the only difference is the details of the GC which are swapped in/out based on a flag. If that's what you're going for, I think that at least provides us a path from the current implementation to allowing the new implementation, enabling by default, deprecating the old one, then removing the old one. However if we have sufficient testing I don't think we necessarily need to go through all of that, we could just stick to one implementation that is incrementally improved/changed.