@user1823 Would you mind reviewing this for me?

I like the improved info for users, but am not very fond of the current approach:

• It runs as the page is loaded, leading to slower loads on large collections
• On change, it's invoking the backend method multiple times for each change. And if the user wants to hold down a key to scroll through dates, it's updating on every change.
• It's using a global cache that isn't cleaned up, and thus leaks memory
• It adds a bunch of extra code that partially duplicates our existing filtering logic, and we'd need to maintain them in lock-step moving forward.
• The user's chosen search also alters how many cards will be shown. This does update the notice when the search is changed, but as they're in different sections, this is not obvious.

What if instead, the optimization routines kept track of included/excluded cards, and returned them at the end of optimization? A message could then be shown as optimization completes indicating the number of cards that were included/excluded. It would mean we can re-use existing codepaths, and the performance impact over what we currently have would be negligible. We already show the review count, so this info could be shown next to it.

WDYT?

So if I understand you correctly, you want to show the number of included and total cards after optimization? I see two problems with that:

1. It's backwards - users set the date first and then run the optimization, in that order.
   "Set the date -> Find out the number of included cards -> Optimize" makes sense
   "Set the date -> Optimize -> Find out the number of included cards" makes much less sense
2. More importantly, it means that if someone wants to see how changing the date affects the number of included cards, he will have to do a lot of optimizations and a lot of scrolling from the FSRS section to the Advanced section and back

• It runs as the page is loaded, leading to slower loads on large collections.

Does this matter if its loaded asynchronously? I don't think it would block the user from interacting with the page while it is loading so it shouldn't matter right?

• On change, it's invoking the back-end method multiple times for each change. And if the user wants to hold down a key to scroll through dates, it's updating on every change.
• It's using a global cache that isn't cleaned up, and thus leaks memory

Another option I was considering would be to instead of calling the back-end several times, have the list of get_cards_first_in_last_group_of passed to the front-end and then use JavaScript to calculate the number of values below the ignored_before. This would fix the memory leak and mean that you would only have to call the back-end once, although I imagine it would make it run slower. Good idea?
We could just clear the cache on page close but that doesn't solve the repeated calls problem

• It adds a bunch of extra code that partially duplicates our existing filtering logic, and we'd need to maintain them in lock-step moving forward.

I'm personally happy with the value being an approximation rather than the exact correct value. Unless there are radical changes to the way ignore-reviews-before works, I can't foresee the SQL becoming terribly inaccurate even if it is ignored when future changes are made.

• The user's chosen search also alters how many cards will be shown. This does update the notice when the search is changed, but as they're in different sections, this is not obvious.

I'm sorry, I can't think of anything that can be done about this 😄.

What if instead, the optimization routines kept track of included/excluded cards, and returned them at the end of optimization? A message could then be shown as optimization completes indicating the number of cards that were included/excluded. It would mean we can re-use existing codepaths, and the performance impact over what we currently have would be negligible. We already show the review count, so this info could be shown next to it.

WDYT?

I could see that working as a compromise. A

100 reviews 
9938/10000 cards ignored by ignored before

like message would probably still prevent people from misconstruing how ignore-reviews-before works. I do personally much prefer the message as it currently exists though. Your call ofc.

Why is calling the backend multiple times an issue? Is that bad coding practise?

Why is calling the backend multiple times an issue? Is that bad coding practise?

On change, it's invoking the back-end method multiple times for each change. And if the user wants to hold down a key to scroll through dates, it's updating on every change.

Oops I miss-read as well. The problem here is that the back-end is being called multiple times every time the value is changed once.

Even with this SQL, we are not being 100% accurate (some cards may have incomplete review history, and thus ignored) and the heavy SQL query is introducing a host of issues (as pointed out by dae). So, I was thinking that we should just switch back to the simple SQL that Luc used initially and then show a message like the following after rounding off the results :

~2,000 / ~10,000 cards will be used to optimize the FSRS parameters.

Edit: Let's not round off the numbers. The error in estimation even with the simple query is likely to be very small and may be even 0. Rounding off will be more harmful than useful. The ~ already suggests that the numbers may not be 100% accurate.

The user's chosen search also alters how many cards will be shown ... but as they're in different sections, this is not obvious.

Is this important? The goal of the PR is just to demonstrate the effect of the ignore cards before option.

I've reverted to a version which has a more approximate but faster SQL (#3910 (comment)) and removed caching. I've also fixed the problem with multiple calls to the back-end. Would this be ok for you?

I like that you've fixed the duplicate calls, and simplified the SQL. If we have to maintain two separate paths, then having one that only tries to be a rough approximation and is not expect to be identical is going to be easier to maintain.

I tested this on a reasonably large collection with a beefy machine from this year, and each call is about 35ms. On slower devices and more extreme collection sizes, it's definitely going to be a noticeable delay. I don't think users should be paying that cost when they don't need it, such as:

• When opening the options window. Unless they're specifically adjusting that option, they likely don't need to see the card count each time. You asked about doing it asynchronously, and that would be an improvement, but I think it would make more sense to wait until the user tries to change the value (or we add an explicit 'check' button).
• When they make edits to the search or limit, the info update should be delayed and only run after, say, 1 second of inactivity, or things may get laggy for users trying to type something in and having it calculated on every keystroke/repeated key.

You asked about doing it asynchronously, and that would be an improvement, but I think it would make more sense to wait until the user tries to change the value (or we add an explicit 'check' button).

Oh I meant that I thought it was already done asynchronously. Does it cause unresponsiveness when you open the deck config?

When opening the options window. Unless they're specifically adjusting that option, they likely don't need to see the card count each time.

Personally, I'm a little concerned that people that have set this option to something wrong before this update will then fail to notice it but It's better than causing lag I suppose. 8ef5d6d

When they make edits to the search or limit, the info update should be delayed and only run after, say, 1 second of inactivity, or things may get laggy for users trying to type something in and having it calculated on every keystroke/repeated key.

Done! 3fd6e82

Thank you for the quick fixes! A gentle reminder about camelCase being the convention.

It doesn't cause a noticeable startup delay on my system, but we have a responsibility not to burn users' CPU cycles unnecessarily. I appreciate your point about helping existing users notice though, and as a compromise, I've temporarily disabled that logic - we can enable it again in a future update.