If we add the following properties we can make things way more arbitrary:

dim_names: dict that maps numerical dim index to a dimension name, ex: {0: "t", 1: "z"}. Only need to name scrollable dims.

index: dict that maps dim name to current index, ex: {"t": 123, "z": 5}

image_dims: tuple[int, int] | tuple[int, int, int] numerical dim indices that define the image displayed in the ImageGraphic or ImageVolumeGraphic.

Also, for window_funcs, if a window func is provided for only one dim then it should apply the window only along that dim. If window_funcs are defined for multiple dimensions, then apply the window for all provided dims. Can also have a way to define the order, ex. do "t" first and "z" etc.

Histogram calculation can also be done within this widget anytime the window funcs or frame_apply funcs are changed. We can then just allow force-setting the HistogramLUTGraph.

Make a few ready-made subclasses of ImageWidgetArray, for example IWA_Dask, IWA_Lazy etc.

In ImageWidget.__init__(), parse each data array in the list use the appropriate IWA type based on the data array type.

IW also has a new kwarg iwa_types which can explicitly define the IWA type which corresponds to the data array.

Make the window functions more flexible. Allow options kwargs that include the full data array, i.e. IWA.data, and the current index. Useful for things like computing a window function and then using that on the original data or current frame. Example, use a rolling filter on the current window, use this to subtract from the original frame.

new kwarg on ImageWidget, "adaptive_histogram", List[bool] | bool which creates an event handler for current_index so that any time the slider is moved or the index is changed it sets the histogram based on the current frame/volume.

If the data doesn't change when moving through dims, example: if there's a simple 2D image amongst multiple txy videos, then would be good to handle this efficiently so that Texture doesn't isn't uploaded when it doesn't change.

ok so another issue is that you can't use a window function when you have a mix of movies and images (no t dim) 😞 . So this separation also needs to fix this issue.

Also need to make it possible for an ImageWidget subplot to be blank, set that data index as None and just ignore when scrolling.

argument to specify a transpose tuple of size 2 or 3 that is applied on the final image/volume after slicing, window functions, etc. could be useful.

ok I think I got a good way to apply window funcs, clean, understandable.

# array with many slider_dims
a = np.random.rand(25, 20, 8, 4, 100, 100)
print("shape of a:\t\t", a.shape)

# final slice is: (i - w, i, i + w)
# specify window size for each dim
windows = (3, None, 2, None)
# order in which window funcs are applied
order = (0, 2)
# window function for each dim
funcs = (np.mean, None, np.std, None)

# current slider indices
indices = (15, 5, 4, 2)

indexer = list()

for i, w in zip(indices, windows):
    if w is not None:
        s = slice(i - w, i + w, 1)  # start, stop, step
    else:
        s = slice(i, i + 1, 1)
    indexer.append(s)

print("indexer:\t\t", indexer)

a_s = a[tuple(indexer)]

# also provide the user the option to use this window-sliced array in one function where they handle everything
print("indexer applied to a:\t", a_s.shape)

for dim in order:
    f = funcs[dim]

    a = f(a, axis=dim, keepdims=True)
    print(f"shape of a after window func: {f} on dim: {dim} is:\t {a.shape}")

Thoughts on how sliders should map onto dims:

if we have this array, let's say it's [t, z, m, n]:
[1_000, 30, 100, 100]

then, do we assume the following smaller array is [t, m, n] or [z, m, n]?
[n, 100, 100]

Another way to think about this:

We want to allow arbitrary numbers of dims, so if we have the following array with ndim 6:
[a, b, c, d, m, n]

[m, n] are the image dims (it could also be RGB or volumetric, that stuff is easy to deal with, ignore it for now)

Do we assume a smaller array with ndim 4 is:
[a, b, m, n]

or:

[c, d, m, n]

Sliders will be made for a, b, c, d so this is important to decide.

One way to thing of it is: right-most are image dims, so slider dims accumulate as soon as image dims finish?
This would mean we interpret the shorter array above as [c, d, m, n]

Another way to think of it is, slider dims accumulate from left to right until they reach the image dims.
This would mean we interpret the above array as [a, b, m, n]

EDIT: We have chosen to do [c, d, m, n]

I think accumulating the slider dims from left to right makes sense.

However, depending on the use-case, I would want to be able to choose between [a b m n] and [c d m n]. For microscopy, [t z m n] is the most popular but I've also seen libraries use [z t m n] (e.g. suite3d). These two dim orders behave completely differently and take a lot of fine-tuning in dask/numpy depending on which you use.

What about defaulting to "left to right" mapping with an optional semantic argument, e.g.:

ImageWidget(
    data=array,
    image_dims=(-2, -1),  # or just always the last two dims
    slider_dims=(0, 1, 2, 3),  # explicit ordering
    dim_names={0: "t", 1: "z", 2: "c"}  # for imgui slider labels
)

Another important thing: other than slider dims, nothing else has to be the same between iw arrays. For example, we can allow each individual array in the ImageWidget to have their own window function or frame apply function. Not sure what's the best API for this though.

Maybe the constructor can take a window funca arg, which if it's a list it defines the window func per data array. This would of course be symmetric with the properties.

display_dims should be a mutable property. When changed from 2 or 3, delete the existing graphic and replace it.

ok I got the basics down!

I now realize the histogram is not gonna be trivial 😄

We can't sub-sample the original array and then pass it through the window funcs because that won't correspond to windows in the original data.

I think we need to do something like this:

• if a window function is used for a dim, use a small number of indices from this dim depending on the window size, but at least 5 or 10. Apply the window function with these indices.
• if a dim has no window function, use some number of indices from this dim, but can be somewhat generous.
• if using a finalizer_func that's applied before displaying the frame or volume, do not subsample these spatial dims, for example a Gaussian kernel would provide very different results on a full spatial sample vs. a subsample.

I think we can use some of the logic in utils.subsample_array and do something like;

1. Create a new func get_subsample_indices that returns the indices necessary to subsample the array.
2. Determine a number of sample indices, call get() with the indices.
3. Concatenate all these arrays and compute the histogram.

@apasarkar @FlynnOConnell

Thoughts on how sliders should map onto dims:

if we have this array, let's say it's [t, z, m, n]: [1_000, 30, 100, 100]

then, do we assume the following smaller array is [t, m, n] or [z, m, n]? [n, 100, 100]

Another way to think about this:

We want to allow arbitrary numbers of dims, so if we have the following array with ndim 6: [a, b, c, d, m, n]

[m, n] are the image dims (it could also be RGB or volumetric, that stuff is easy to deal with, ignore it for now)

Do we assume a smaller array with ndim 4 is: [a, b, m, n]

or:

[c, d, m, n]

Sliders will be made for a, b, c, d so this is important to decide.

One way to thing of it is: right-most are image dims, so slider dims accumulate as soon as image dims finish? This would mean we interpret the shorter array above as [c, d, m, n]

Another way to think of it is, slider dims accumulate from left to right until they reach the image dims. This would mean we interpret the above array as [a, b, m, n]

EDIT: We have chosen to do [c, d, m, n]

ok so both are actually easy to do and this is something that ImageWidget can handle before calling get() on each of the individual NDImageViews:

# consider an example index from all 4 sliders
indices = (100, 15, 5, 3)

# if we have 4 example array which have the following number of slider dims
n_slider_dims = [4, 3, 2, 1]

right -> left ordering

# get the indices we use for each of the 4 arrays with different numbers of dims
for n in n_slider_dims:
    print(f"{n} slider_dims, right -> left indices are: {indices[-n:]}")

out:

4 slider_dims, right -> left indices are: (100, 15, 5, 3)
3 slider_dims, right -> left indices are: (15, 5, 3)
2 slider_dims, right -> left indices are: (5, 3)
1 slider_dims, right -> left indices are: (3,)

left -> right ordering

for n in n_slider_dims:
    print(f"{n} slider_dims, left -> right indices are: {indices[:n]}")

out:

4 slider_dims, left -> right indices are: (100, 15, 5, 3)
3 slider_dims, left -> right indices are: (100, 15, 5)
2 slider_dims, left -> right indices are: (100, 15)
1 slider_dims, left -> right indices are: (100,)

@FlynnOConnell

my all time favorite commit in fastplotlib 😄 : 7770ee0

beauty lol

ok the basics work! 🥳

Needs tests, more manual testing, some cleanup and tweaks and will be ready to go! I anticipate there will be a few growing pains, lots of parsing since this has to deal with a diverse range of complex datasets and compute possibilities so it may take a while to iron out all the bugs.

@apasarkar @FlynnOConnell @clewis7 gonna need a review from all of you :D

Should make ImageWidgegt.histogram_widget settable, and it should update the NDImageProcessors accordingly and reset the histogram graphics.

Need to implement ImageWidget properties and setters for window func stuff and finalizer func. Each of these should call ImageWidget._reset

OK can get and set data, all window function stuff, and finalizer_func, per-data array using the index or subplot name. Likewise dimension indices can be get or set on a per-dimension basis.

So all this works:

iw.data[2] = new_array_of_any_dims

# can also use the subplot name
iw.data["pmd"] = new_pmd_array

# likewise for other image processor properties
iw.window_funcs[1] = (None, np.mean)  # mean window function only on dim at index 1
iw.window_sizes[1] = (None, 23)

# similar for dimension indices
iw.indices[1] = 7

# if `slider_dim_names` is set, can also use that:
iw.slider_dim_names = ("t", "z")
iw.indices["t"] = 5

otherwise you'd have to do this (you can also do this if necessary)

# set all new data arrays
iw.data = [new_a, new_b]

# set only the 2nd one as new
iw.data = [old_a, new_b]

# this is equivalent to
iw.data[1] = new_b

# same for all other window funcs

# likewise for setting dim indices
# if the current indices are already (10, 0)
iw.indices = (10, 3)  # keep first dim same, change second dim
# the above is identical to this:
iw.indices[1] = 3

@FlynnOConnell

There is a new requirement that the data input must implement astype, due to /graphics/features/_image.py calling _fix_data(), which it wasn't before?

If so, might be helpful to add something like

if hasattr(data, "astype"):

else:
("Input data must implement astype") 
...

to the _is_arraylike()?

Histogram stuff I tested which all works:

• all same vals, all zeros, nans, infs

Also tested large ranges such as between 0 - 2^16, 0 - 2^32, 0 - 2^64,
also: 1 / 2^16 - 2^16, -2^16 - 2^16, tested 2^32 as well with all these.
@apasarkar

Can also have a mode where the same histgram lut tool just shows the colorbar with the scalebar and the linear region selector can change the vmin, vmax, probably in a future PR.

I encountered what may be a bug while making the mcorr widget. NDImageProcessor.get() was using a window function on the mcorr array which has dims [t, m, n], and then getting an array of shape (170,) which is really weird. I can't seem to easily reproduce this though so not sure how this happened.

We should also try to support arrays with dynamic shapes, i.e. where the shape of the array is a function of one of the indices of a dimension 😆 .

This may sound weird but it has a very clear usecase, multi-session data! @apasarkar @FlynnOConnell

For example consider a multi session array with the following shape:

[s, t, z, m, n]: [session, time, z-plane, m, n]

Each session may have a different number of timepoints or z planes!

I think we can actually implement this very easily in a future MultiSessionNDImageProcessor subclass by just making the shape property something like this:

@property
def shape(self) -> tuple[int, ...]:
  match self._last_indices  # last indices tuple that was used on a `get()` call
    case 0: # session 0
      return ... # shape of session 0
    ...

The issue with this is that we've written NDImageProcessor to specifically have no notion of storing a slice index. The ImageWidget is responsible for maintaining the current slice indices and the NDImageProcessor just processes and gives it to ImageWidget. Can think later about the best way to make this work.

Maybe when get() is called, it uses the session dimension to set the current shape? Will need to see if dropping this in works directly.