JAMES-3340 JMAP thread collapse option for Email/query #2792
Conversation
| public Flux<MessageId> listMailboxContentSortedBySentAt(MailboxId mailboxId, Limit limit, boolean collapseThreads) { | ||
| Preconditions.checkArgument(!limit.isUnlimited(), "Limit should be defined"); | ||
|
|
||
| Limit limitFetch = defineLimitFetch(limit, collapseThreads); |
There was a problem hiding this comment.
Cassandra is defo trickier :)
I'm not sure here...
I've been thinking quite a lot, this is a proposition among others. I feel like looping on cassandra requests until we meet the limit for collapseThreads would complexify greatly the code and for not such a great gain (more costly in perf too).
I thought:
- or we do like some requests that need to be post-sorted, just fetch all records, then collapsing threads and take the limit
- or we fetch an upper limit (here 3 times the limit defined) and we collapse threads and then take up to the limit. We accept that maybe sometimes we could have a bit less than the limit (hopefully rare enough) but it should stay balanced in terms of perf and keep the code relatively simple.
I tried to look at paging too but that would complexify a lot the code as well I believe (like looping).
Well brain storming welcomed honestly
There was a problem hiding this comment.
I can get a try to the cassandra implem but yes we would need some extra work for Cassandra.
However can we focus on the short term on SaaS related topis?
…with EmailQueryView
148c930 to
5bebc68
Compare
…ndling with EmailQueryView
|
Cassandra implem passes the tests... Just needs a good refactoring to make something really clean (cause quite dirty atm) and still left the pg implem. |
| if (collapseThreads) { | ||
| return baseEntries.collectList() | ||
| .flatMapMany(results -> { | ||
| List<EmailEntry> distinctByThreadId = distinctByThreadId(results); | ||
| boolean hasEnoughResults = distinctByThreadId.size() >= limit.getLimit().get(); | ||
| boolean isExhaustive = results.size() < backendFetchLimit.getLimit().get(); | ||
| if (hasEnoughResults || isExhaustive) { | ||
| return Flux.fromIterable(distinctByThreadId) | ||
| .take(limit.getLimit().get()) | ||
| .map(EmailEntry::getMessageId); | ||
| } | ||
| Limit newBackendFetchLimit = Limit.from(backendFetchLimit.getLimit().get() * COLLAPSE_THREADS_LIMIT_MULTIPLIER); | ||
| return listMailboxContentSortedBySentAtWithBackendLimit(mailboxId, limit, collapseThreads, newBackendFetchLimit); | ||
| }); |
There was a problem hiding this comment.
We need an abstraction not to repeat this piece of code.
Refactorig?
Still left Cassandra and Postgresql implem finalizing