perf: Use flyweight for node fields #1654
Conversation
bigframes/core/chain_list.py
Outdated
| @@ -0,0 +1,53 @@ | |||
| # Copyright 2024 Google LLC | |||
bigframes/core/chain_list.py
Outdated
| # * Support insertions and deletions | ||
|
|
||
|
|
||
| class ChainList(collections.abc.Sequence[T]): |
There was a problem hiding this comment.
We should add some more doc to describe what this classes does. It's not obvious that the goal is to quickly locate an element from a concatenated sequence.
Plus, perhaps "ChainedSequence" is a better name?
In addition, this implementation might hurt performance if the provided "parts" are all very short. In that case, we are effectively using iterations in getitem(). Does it make sense to have certain kind of threshold? That is, if the total number of element is below certain threshold (10 maybe?), just concat everything into a tuple. Otherwise, use the current technique.
There was a problem hiding this comment.
added docstring and renamed
Added some conservative compaction logic as well. Key here is memory efficiency/reuse rather than lookup efficiency, so don't want to go to wild on those types of optimizations.
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> 🦕