🦋 Changeset detected

Latest commit: abe6b37

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

_{tachometer-reporter-action v2 for Benchmarks}

PS - Looks like there was supposed to be a PR template, but the field was just filled with "...LL error" for me

I think that was just the end of your commit message

This looks like a great change to me. I'd like to give Gray a chance to review in case there's some subtlety where we'd rather fail on a missing controller here

Oh, please add a changeset by running npm run changeset at the root of the monorepo

These changes may indeed be good, but if possible I'd like to understand better how it is that the virtualizer in HomeAssistant ends up in a state where it has no ScrollerController but is trying to correct a scroll error, and ideally I'd like to come up with a minimal repro case so that we can write a test or two.

My guess is that the virtualizer is being disconnected somewhere in the chain of events and then subsequently responds to an async scroll error message (perhaps triggered by a MutationObserver callback) that no longer applies. If this is the case, there may be a spot earlier in the codepath where we can short-circuit, rather than getting all the way into _correctScrollError() without a controller in place.

It's originating from the children resize observer, and it certainly occurs only after a disconnect. It's possible the items get removed before the virtualizer element is fully disconnected (and maybe that observer isn't considering that situation?). I don't quite understand that callback enough at the moment and probably not the best to come up with a minimal test. Here's the longer stack trace:

Uncaught (in promise) TypeError: this._scrollerController is null
_correctScrollError
Virtualizer.ts:741

_finishDOMUpdate
Virtualizer.ts:546

_updateDOM
Virtualizer.ts:538

_handleLayoutMessage
Virtualizer.ts:603

_layout
Virtualizer.ts:441

_sendStateChangedMessage
BaseLayout.ts:478

_reflow
BaseLayout.ts:364

_reflow
flow.ts:491

reflowIfNeeded
BaseLayout.ts:257

_updateLayout
Virtualizer.ts:563

_schedule
Virtualizer.ts:520

_measureChildren
Virtualizer.ts:501

_childrenSizeChanged
Virtualizer.ts:888

_initObservers
Virtualizer.ts:275

connected
Virtualizer.ts:287

_makeVirtualizer
virtualize.ts:130

_initialize
virtualize.ts:141

update
virtualize.ts:91

Given that, I suppose another solution would be to change the order in disconnected(), so that the observers are disconnected before _scrollerController is detached, but maybe that's a distinction without much of a difference?

Thanks for the additional info, @steverep. I'll play around a bit and see if I can come up with a minimal repro, though I may have to come back to you with followup questions.

@steverep I expect to have time to dig into this later today, but two quick questions:

1. Does this happen to be reproducible in the Home Assistant demo deployment, where I can easily see it?
2. Would it be easy to point me to the relevant source files in the Home Assistant repo?

1. Does this happen to be reproducible in the Home Assistant demo deployment, where I can easily see it?

Unfortunately, it is not. Most if not all the virtualizers are in the settings area which isn't in the demo.

2. Would it be easy to point me to the relevant source files in the Home Assistant repo?

Yes, of course. The repro I am using comes from the virtualizer in our ha-data-table component. Sorry, it's one of the more complicated components though.

Essentially, the repro goes like this:

1. Scroll it
2. Click a row which brings you to another page in the SPA (removes the virtualizer from DOM)
3. Click another link in the new page to navigate again, then error is in the console

Let me know if I can test anything for you.

@cintaccs In your case (as described in #4238), can you explain how and why the virtualizer is being destroyed / disconnected when the data provided by the containing view changes? I would expect the virtualizer to rerender, but not disconnect.

FWIW, I took a closer look at this today. If I'm reading the ResizeObserver spec correctly, calling disconnect on the RO should theoretically guarantee the callbacks have all returned since everything is done in the event loop. That doesn't include anything spawned asynchronously of course, which the _childrenRO does.

The deeper problem seems to be that in _finishDOMUpdate, which is within the asynchronous part of the callback, _childrenRO.observe() is being called. So the RO is is disconnected, only to just start observing again if that stuff hasn't finished. Indeed if I set this._childrenRO = null after disconnecting, I get the following error instead:

Uncaught (in promise) TypeError: this._childrenRO is null
    _finishDOMUpdate Virtualizer.ts:545
    _finishDOMUpdate Virtualizer.ts:545
    _updateDOM Virtualizer.ts:542
    _handleLayoutMessage Virtualizer.ts:605
    _layout Virtualizer.ts:449
    _sendStateChangedMessage BaseLayout.ts:478
    _reflow BaseLayout.ts:364
    _reflow flow.ts:491
    reflowIfNeeded BaseLayout.ts:257
    _updateLayout Virtualizer.ts:566
    _schedule Virtualizer.ts:524
    async*set items Virtualizer.ts:252
    _updateVirtualizerConfig virtualize.ts:107
    update virtualize.ts:89
    _$resolve directive.ts:134
    resolveDirective lit-html.ts:1083
    _$setValue lit-html.ts:1362
    _update lit-html.ts:1192
    _commitTemplateResult lit-html.ts:1540
    _$setValue lit-html.ts:1384
    render lit-html.ts:2183
    update lit-element.ts:165
    performUpdate reactive-element.ts:1329
    scheduleUpdate reactive-element.ts:1261
    __enqueueUpdate reactive-element.ts:1233
    requestUpdate reactive-element.ts:1208
    set reactive-element.ts:726
    _commitValue lit-html.ts:1883
    _$setValue lit-html.ts:1825
    _update lit-html.ts:1192
    _commitTemplateResult lit-html.ts:1540
    _$setValue lit-html.ts:1384
    _update lit-html.ts:1192
    _commitTemplateResult lit-html.ts:1540
    _$setValue lit-html.ts:1384
    render lit-html.ts:2183
    update lit-element.ts:165
    performUpdate reactive-element.ts:1329
    scheduleUpdate reactive-element.ts:1261
    __enqueueUpdate reactive-element.ts:1233
    requestUpdate reactive-element.ts:1208
Virtualizer.ts:545

Note the RO callback is not in the trace this time. It's coming from an earlier call to update the DOM.

@steverep Yes, I believe the problem is that something in application code (probably related to the mechanism for switching between views as the user navigates) is causing the virtualizer to kick off an update cycle after it has been disconnected.

In trying to create a minimal repro was, I have tried various ways of toggling between a view with a virtualizer and a view without a virtualizer. However, the only way I've been able to end up in _correctScrollError without a ScrollerController present is really contrived: I intentionally trigger a virtualizer update (by setting its items property to a different value) at the same time as I'm triggering the view update that will cause the virtualizer to be disconnected.

The following Playground example illustrates:

https://lit.dev/playground/#gist=e18207161b9d96c5d81dfb7032f474ad

If you run it as-is (with the Console open to view log statements) and click the Toggle button, everything is fine. But if you uncomment line 35 and then toggle, you'll see we end up in _correctScrollError without a controller.

The good news is that it should probably be fairly straightforward to short-circuit the virtualizer update cycle in the case where it is not currently connected.

However, I would still like to see if we can figure out what it is in the Home Assistant code (and in @cintaccs app) that triggers a virtualizer update after disconnection so that we can make some more realistic tests.

I do see that there's a set items in your stack trace. Do you know why that would be happening after (or concurrently with) the view change that causes the virtualizer to disconnect?

@graynorton based on my testing, I would suggest you set this._childrenRO = null per my last reply. I find that the problem is much more reproducible after doing that.

In trying to create a minimal repro was, I have tried various ways of toggling between a view with a virtualizer and a view without a virtualizer. However, the only way I've been able to end up in _correctScrollError without a ScrollerController present is really contrived: I intentionally trigger a virtualizer update (by setting its items property to a different value) at the same time as I'm triggering the view update that will cause the virtualizer to be disconnected.

Yeah, I think that's more or less what is happening.

I do see that there's a set items in your stack trace. Do you know why that would be happening after (or concurrently with) the view change that causes the virtualizer to disconnect?

Nope, the only thing I've narrowed down is that it only seems to error when we have the data filtered, and that means a web worker is involved in returning the reactive property that holds the items. Although I don't immediately see why that would be a problem.

I'll amend my last post with the full stack trace. It actually seems to be looping inside lit for a couple update cycles, so maybe the items were actually set long before? 😕

The good news is that it should probably be fairly straightforward to short-circuit the virtualizer update cycle in the case where it is not currently connected.

I'm looking at how to implement that, but how far upstream which would still be safe is not obvious to me. Certainly stopping at the start of this._updateDOM would work, but are you thinking further?

@cintaccs In your case (as described in #4238), can you explain how and why the virtualizer is being destroyed / disconnected when the data provided by the containing view changes? I would expect the virtualizer to rerender, but not disconnect.

The virtualizer is within a child component of the page that is being refreshed. In order to ensure that the content is really refreshed one of the parents are re-created. In some earlier lit versions at least - this was the only way I could figure out to get the page to refresh. Later the virtualizer was introduced in the child list component. (where the list component is a child of the page component...).

From my point of view - I wouldn't consider it bad to check for existence before destroying .... I guess destroy could happen for a lot of difference reasons - navigating away etc.

@steverep My apologies—of course! The screenshot was a dumb way to share the diff in the first place.

diff --git a/packages/labs/virtualizer/src/Virtualizer.ts b/packages/labs/virtualizer/src/Virtualizer.ts
index f7fb07c8..7ba732f8 100644
--- a/packages/labs/virtualizer/src/Virtualizer.ts
+++ b/packages/labs/virtualizer/src/Virtualizer.ts
@@ -549,8 +549,16 @@ export class Virtualizer {
     }
   }
 
+  _shouldUpdateLayout() {
+    // Only update the layout and trigger a re-render if we have...
+    //   a) A layout
+    //   b) A scrollerController, which means we're connected
+    //   c) An offsetParent, which means we're not hidden
+    return this._layout && this._scrollerController && this._hostElement?.offsetParent;
+  }
+
   _updateLayout() {
-    if (this._layout) {
+    if (this._shouldUpdateLayout()) {
       this._layout!.items = this._items;
       this._updateView();
       if (this._childMeasurements !== null) {

@graynorton I can test with HA later tonight and let you know.

Regarding no other changes, correct me if I'm missing something, but wouldn't the changes in this branch still be necessary as a safeguard? Your change may catch most issues, but it doesn't guarantee the timing, i.e. a disconnect can still happen after that checkpoint.

@steverep I don’t believe so, if everything is working as designed. The fact there is no guard code there is actually intentional, so we don’t inadvertently mask more serious issues, like the unnecessary render that #4182 revealed.

We can and should still remove the truly redundant non-null assertions you noticed in other spots, though.

@steverep I’ll add that I’m very much open to being proven wrong, but before we go and guard against it on a blanket basis, I’d really like to have a concrete, understandable repro that shows how a disconnection might occur in the middle of a virtualizer update cycle. I believe the timing of the mutation and resize observers makes this unlikely or impossible, but I’m not 100% sure.

@graynorton I tested HA last night and was unable to reproduce the error with just your change. 👍🏻

I'll admit my gut is still telling me the timing cannot be guaranteed once _updateDOM fires, but I'll defer to your judgement as you obviously understand the code much better than me. Time will tell anyway, because if there's a way to break it, HA users will find it. 😉

To that end, I reverted the controller and observer checks here. The only thing I kept (other than the removal of non-null assertions) was nullifying the the observers on disconnect. I would argue that's still a worthwhile change as, to your point, it would cause earlier failures after a disconnect by making sure re-observations cannot occur (as was happening here with _childrenRO).

Do you want to add your change here or shall I change the commit message to reflect it as is?

@cintaccs thank you for your response!

The virtualizer is within a child component of the page that is being refreshed. In order to ensure that the content is really refreshed one of the parents are re-created. In some earlier lit versions at least - this was the only way I could figure out to get the page to refresh. Later the virtualizer was introduced in the child list component. (where the list component is a child of the page component...).

From what you've reported, it seems most likely that the "old" virtualizer (the descendant of the ancestor view that is being replaced) is somehow receiving property updates that trigger it to re-render, despite no longer being present in the document's DOM tree. This is a case we should be handle better regardless, but I would still like to understand as clearly as I can the patterns that cause it to occur and write minimal but realistic tests representing these patterns. With that goal in mind, here are a couple of follow-up questions for you:

• Am I understanding correctly that the ancestor element is not just re-rendered, but actually destroyed and recreated? How is this achieved?
• By what means does the virtualizer receive its data? For instance, is the items property simply populated by properties flowing down the tree of rendered components, or is the virtualizer (or a detached ancestor) subscribed to async updates via some other means?

hmm.. I wish it was simple to answer.
The challenge here is that in the original design of components (early lit days) I had issues triggering a re-render when switching views on the same data model (i.e. switching from cards to list view on the same data - OR - switching in-between different sets of filters for the listings). Also the UI is created to not flicker blank white pages so the header will stay the sub-header might stay - the body of the list will either refresh or be replaced with another type of view etc.

In general I am using the properties of the lit element class components - normal use I would think - however some of these properties are either ARRAYs or OBJECTS and also might be delayed with a event UP and await down flow...

The virtualizer is not subscribing in itself to this - it just knows the items array ...

Thanks, @steverep!

To that end, I reverted the controller and observer checks here. The only thing I kept (other than the removal of non-null assertions) was nullifying the the observers on disconnect. I would argue that's still a worthwhile change as, to your point, it would cause earlier failures after a disconnect by making sure re-observations cannot occur (as was happening here with _childrenRO).

Agreed!

Do you want to add your change here or shall I change the commit message to reflect it as is?

I am flexible on this. I guess it might be somewhat simpler to go ahead and land your improvements first, then have me make a follow-up PR with my change (and likely also some new tests)—only because that way, we don't need to worry about permissions for me to push to your fork, etc.

Another possibility would be for you to just go ahead and apply my patch to your fork so that it ends up here; that would be fine with me if it works for you. Just let me know what you prefer!

Regardless, I personally won't have much time with my computer for the next few days, as I'm about to head out on a trip. I'm hoping there will be some downtime that I can spend to advance this, but in the worst case it would be next Tuesday or Wednesday before I was able to get back to it. I should be able to keep my eye on the discussion here, though.

@cintaccs

hmm.. I wish it was simple to answer.

No worries! Thanks for the additional background. I suspect that the changes Steve and I have identified in this PR will fix the issue for you as well. If it would be feasible for you to try applying my patch to your local setup, that would be great. But if not, that's fine—you can wait until the changes land and let us know if you're still seeing the problem then.

@graynorton I went ahead and applied your patch with a minor refactoring because your "should" method confusingly didn't return a boolean and it would just force back more non-null assertions. Rather than adjust that I just incorporated the check into _updateLayout. Hope that's okay.

If so, I suppose this is ready to go.

Whenever this is merged and released, it's now going to be difficult to incorporate without first upgrading to lit 3.0. The choice to only have a patch release 2.0.8 after updating the lit dependency doesn't make sense to me.

@steverep

The choice to only have a patch release 2.0.8 after updating the lit dependency doesn't make sense to me.

Can you explain what you mean here? Making this a minor release wouldn't help, would it?

Can you explain what you mean here? Making this a minor release wouldn't help, would it?

No, this PR should definitely only be a patch. My point was that the just released 2.0.8 now has a direct dependency on lit 3.0, with no inclusion of 2.x versions. That makes it a breaking change IMO, and the new virtualizer version should have been 3.0 (unless it still supports lit 2.0 in which case the manifest should be updated).

Right now, you cannot update to 2.0.8 without also updating lit to 3.0 unless you bundle 2 different lit versions which I wouldn't dare do (home-assistant/frontend#18206).

That makes it a breaking change IMO, and the new virtualizer version should have been 3.0 (unless it still supports lit 2.0 in which case the manifest should be updated).

Agreed. I went ahead and merged this PR; will make a separate one to update the version range for the lit dependency.