I would expect this is not needed anymore, and actually doesn't seem used. This used to be to ensure we didn't move onto generating a fetch before retrieving positions to commit (both steps were independently triggered from the app thread), but that's not the case anymore right?)

I'd noticed that the ShareConsumerImpl was still using PollEvents, so I didn't want to muck with it too much. It seems like I could refactor processPollEvent() into two separate pieces of logic, one for the CompositePollEvent and one for the (Share consumer) PollEvent.

Since ShareConsumerImpl doesn't use the Future from the event, I'll remove it and clean that up.

Done.

I see this hasn't been addressed. Seems like unneeded states/transitions that make the CompositePollEvent more complex, but I could be missing why we may need it like this?

this logic is to maybe jump back to the app thread, but at this point we haven't really done anything in the background yet, so couldn't we avoid this hop and check for this in the app thread directly?

maybeCompleteExceptionally() checks NetworkClientDelegate.getAndClearMetadataError(). We can't access that from the application thread, so this is the first available opportunity to do so.

It's possible for there to be offset commits that finished after the last AsyncPollEvent completed, so maybeCompleteWithCallbackRequired() catches those cases before we've started any new work on the new poll.

maybeCompleteExceptionally() checks NetworkClientDelegate.getAndClearMetadataError(). We can't access that from the application thread, so this is the first available opportunity to do so.

I was thinking an atomic variable. Metadata errors are discovered in the background indeed, and we need to read it from the app thread on poll (and clear it I guess). Maybe an atomic var shared via applicationEventHandler, for instance? (ApplicationEventHandler lives in the AsyncConsumer, and has ref to the network component, seems like the component in-between). The network layer sets the atomic var, we read it on poll and throw if there is anything there. Would that work? If it works, we could solve the inter-thread communication for this case with it, instead of jumping back and forth between the threads.

It's possible for there to be offset commits that finished after the last AsyncPollEvent completed, so maybeCompleteWithCallbackRequired() catches those cases before we've started any new work on the new poll.

Agree that we may have commits that completed after the last AsyncPollEvent, but why do we need to go to the background to trigger them? I was imagining if we could just trigger commit callbacks in the app thread before triggering the (next) AsyncPollEvent

this should be removed (referred to the markReconcileAndAutoCommitComplete that was removed)

Done.

Totally agree with maybeCompleteWithCallbackRequired here (right after triggering a reconciliation, we may have callbacks to run)

But I'm still not convinced about the need for maybeCompleteExceptionally here. That one is to propagate metadata errors we may have received in MetadataReponses. The classic consumer would throw them on the first call to client.poll hit from the consumer.poll (could be dealing with the coordinator, or later on). With the async consumer, I would imagine it should be enough to have a single point within the consumer.poll where we "maybeThrowMetadataErrors"

We already have an initial check to "maybeThrowMetadataErrors" (where the comment is to maybe move to the app thread even). Shouldn't that be enough?

I don't believe we can introduce a maybeThrowMetadataErrors step in the application thread as it needs to access the NetworkClientDelegate owned by the background thread, plus it would need to be a blocking call.

I will experiment with leaving the first call to maybeCompleteExceptionally() in the process() method, and removing the subsequent checks.

I don't quite get why we need this whole new state to trigger rebalance/commit callbacks?

We already have thread-safe queues where elems are added IF there are callbacks that need to be executed, so I was expecting that we simply need to check if there is something on those queues and run it at the beginning of each poll iteration?

    public void poll(Timer timer) {

        // trigger rebalance and commit callbacks if any - what applicationThreadCallbacks.run() does
        offsetCommitCallbackInvoker.executeCallbacks();
        processBackgroundEvents();

        if (inflight == null) {
            log.trace("No existing inflight event, submitting a new event");
            submitEvent(ApplicationEvent.Type.POLL, timer);
        }
        ....

Would that work?

In the current implementation, the commit/rebalance callbacks are executed on the application thread after the PollEvent and before the CheckAndUpdatePositionsEvent. The CALLBACKS_REQUIRED state in this implementation is used to exit from the background thread at that in-between point to effectively achieve the same timing result.

I've made a few attempts to move the callbacks to execute before the events and remove the state, as it is clunky. However, doing so fundamentally changes the established timing. This is further reflected by the fact that multiple unit/integration tests start to fail when the timing is changed by removing the state.

this whole flow is already looking much simpler, nice! but still, this intermediate state seems to be only to ensure we don't attempt to trigger reconciliations/callbacks/metadataErrors on the first poll after the one where one of those were triggered, correct? But do we really need that? It would mean that we would skip triggering a reconciliation when it's maybe needed (would happen on a following poll, but why skipping it in the current one?)

Alternatively, I imagine we could simply trigger all actions on each poll (trigger reconciliation, commit, updateFetchPositions, trigger fetch), short-circuiting as needed back to app thread, but no need to carry on from where we left of given that is all about triggering actions that will simply be no-op if repeated and not needed (and if we end up triggering reconciliation/callbacks/errors on 2 subsequent polls, it's because it's indeed needed)

All those steps have their own "flow control" btw (maybeReconcile will only do something if no other reconciliation going on, metadata errors are cleared when thrown, auto-commit considers the interval and inflights...) that's why I wonder if we can just simply trigger them all on each iteration, no more state machine on the ASYNC_POLL?

maybeCompleteExceptionally() checks NetworkClientDelegate.getAndClearMetadataError(). We can't access that from the application thread, so this is the first available opportunity to do so.

I was thinking an atomic variable. Metadata errors are discovered in the background indeed, and we need to read it from the app thread on poll (and clear it I guess). Maybe an atomic var shared via applicationEventHandler, for instance? (ApplicationEventHandler lives in the AsyncConsumer, and has ref to the network component, seems like the component in-between). The network layer sets the atomic var, we read it on poll and throw if there is anything there. Would that work? If it works, we could solve the inter-thread communication for this case with it, instead of jumping back and forth between the threads.

It's possible for there to be offset commits that finished after the last AsyncPollEvent completed, so maybeCompleteWithCallbackRequired() catches those cases before we've started any new work on the new poll.

Agree that we may have commits that completed after the last AsyncPollEvent, but why do we need to go to the background to trigger them? I was imagining if we could just trigger commit callbacks in the app thread before triggering the (next) AsyncPollEvent