fix: dendrogram edgecases #6669

hoxbro · 2025-08-25T15:22:31Z

List of changes, a missing checkmark means a missing unit test

Handle gridded dataset.
Improve linkage, not being able to solve error message
Fix sorting of non-selected column
Fix using non-plotting vdims as main_dim
Improve error message using Layout with dendrograms.

codecov · 2025-08-25T15:46:06Z

Codecov Report

❌ Patch coverage is 92.59259% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.02%. Comparing base (365b4a4) to head (434d07e).

Files with missing lines	Patch %	Lines
holoviews/operation/element.py	85.71%	4 Missing ⚠️
holoviews/tests/operation/test_operation.py	95.23%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6669   +/-   ##
=======================================
  Coverage   89.02%   89.02%           
=======================================
  Files         329      329           
  Lines       70422    70489   +67     
=======================================
+ Hits        62693    62754   +61     
- Misses       7729     7735    +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

flying-sheep · 2025-08-26T08:34:55Z

Not entirely sure why A.obs["n_counts"] does not have the same size even though A.obs.index does.

because of this line; only kdims currently get expanded (and expand_grid_coords only supports kdims):

https://github.com/holoviz-topics/hv-anndata/blob/b399744dfc8f01b7be159a738cd0007481d1e338/src/hv_anndata/interface.py#L477

flying-sheep · 2025-08-26T10:23:05Z

best wait until holoviz-topics/hv-anndata#82 is fixed

flying-sheep · 2025-08-28T10:48:48Z

OK, so values should now works as expected in holoviz-topics/hv-anndata#66 and broadcast things correctly.

But when using that PR and this one together, things are still broken (in fact, this PR does nothing to change what happens here):

import holoviews as hv
import numpy as np
import scanpy as sc
import hv_anndata
from hv_anndata import ACCESSOR as A
from hv_anndata import register

register()
hv.extension("bokeh")


markers = ["C1QA", "PSAP", "CD79A", "CD79B", "CST3", "LYZ"]
hm = hv.HeatMap(
    adata[np.argsort(adata.obs["bulk_labels"], stable=True), markers],
    [A.obs.index, A.var.index],
    [A[:, :], A.obs["n_counts"]],
).opts(xticks=0, colorbar=True, width=500, height=200)
if dendrogram:
hm = hv.operation.dendrogram(
    hm,
    adjoint_dims=[A.obs.index],
    main_dim=A.obs["n_counts"],
    linkage_metric="euclidean",
)
hm

running with dendrogram = False

running with dendrogram = True

old

flying-sheep · 2025-08-28T14:04:53Z

I updated the image. Looks better, but still changed. Looks reordered, but dendrogram’s optimal_ordering is False by default.

(and the dendrogram is still a little messed up. Very messed up when passing responsive=True)

hoxbro · 2025-08-28T15:06:24Z

I updated the image. Looks better, but still changed. Looks reordered, but dendrogram’s optimal_ordering is False by default.

I think it should be reordered, it is finding an order, just not the optimal order. However, I'm not an expert in dendrograms by any means.

(and the dendrogram is still a little messed up. Very messed up when passing responsive=True)

Can you share an image. The responsive=True is tracked here, #6527

flying-sheep · 2025-08-29T13:01:56Z

OK, two issues:

If you put the dendrogram on the other axis, the ordering is still not preserved:

markers = ["C1QA", "PSAP", "CD79A", "CD79B", "CST3", "LYZ"]
hm = hv.HeatMap(
    adata[np.argsort(adata.obs["bulk_labels"], stable=True), markers],
    [A.obs.index, A.var.index],
    [A[:, :], A.var["n_counts"]],
).opts(xticks=0, width=500, height=400)

hv.operation.dendrogram(
    hm,
    adjoint_dims=[A.var.index],
    main_dim=A.var["n_counts"],
    linkage_metric="euclidean",
)

adjoint_dims=[A.obs["bulk_labels"]] doesn’t work, as what comes out of its groupby can’t be np.vstacked. I’m trying to reproduce scanpy’s heatmap:

Can you share an image. The responsive=True is tracked here, #6527

with the code from my first comment, but responsive=True or frame_width=...:

hoxbro · 2025-08-29T14:04:14Z

If you put the dendrogram on the other axis, the ordering is still not preserved:

I think this occurs because of the difference between np.unique (sorting) and pd.unique (order of occurrence)

import holoviews as hv
import numpy as np
import pandas as pd
import scanpy as sc

import hv_anndata
from hv_anndata import ACCESSOR as A
from hv_anndata import register

register()

hv.extension("bokeh")

adata = sc.datasets.pbmc68k_reduced()

kdims = [A.obs.index, A.var.index]
vdims = [A[:, :], A.obs["n_counts"]]


ds = hv.Dataset(adata[:10, :10], kdims=kdims, vdims=vdims)
hm = hv.HeatMap(ds)
de = hv.operation.dendrogram(
    hm,
    adjoint_dims=[A.obs.index],
    main_dim=A.obs["n_counts"],
    linkage_metric="euclidean",
)
(hm + de).opts(shared_axes=False)

df = ds.dframe()
pd.unique(df["A.var.index"])
np.unique(df["A.var.index"])

with the code from my first comment, but responsive=True or frame_width=...:

Did you have a problem with sizing outside responsive=True / frame_width ?

flying-sheep · 2025-08-29T15:35:56Z

Did you have a problem with sizing outside responsive=True / frame_width ?

nope!

flying-sheep · 2025-09-08T12:27:50Z

Now there’s no exception, but also no dendrogram. Are you testing this with Basic.ipynb in hv-anndata?

hoxbro · 2025-09-08T12:34:38Z

Now there’s no exception, but also no dendrogram. Are you testing this with Basic.ipynb in hv-anndata?

I'm currently just looking at "pure" pandas and trying to tackle point 1 you raised. Point 2, I'm not sure if it is currently feasible, and will therefore likely not be part of this fix PR. I think this is what you are seeing by using bulk_labels.

flying-sheep · 2025-09-08T13:16:00Z

OK, cool! I filed #6683 for that

Otherwise, things seem to work when using adjoint_dims=[A.obs.index] with this PR

hoxbro · 2025-09-08T13:43:03Z

I was about to write that I would file it, but then I saw something weird.

There appears to be a transpose issue when using anndata with HeatMap. Do you have an idea why? Hovering over the data, it matches up with the DataFrame.

Code

import holoviews as hv
import numpy as np
import scanpy as sc

import hv_anndata
from hv_anndata import ACCESSOR as A
from hv_anndata import register

register()

hv.extension("bokeh")

adata = sc.datasets.pbmc68k_reduced()
adata.obs.index = map(str, range(len(adata.obs.index)))  # Just for my own sake...

kdims = [A.obs.index, A.var.index]
vdims = [A[:, :]]

ds = hv.Dataset(adata[:10, :10], kdims=kdims, vdims=vdims)
hm_pd = hv.HeatMap(ds.clone(data=ds.dframe())).opts(tools=["hover"], title="pandas")
hm_adata = hv.HeatMap(ds).opts(tools=["hover"], title="anndata")

(hm_adata + hm_pd).opts(shared_axes=False)

flying-sheep · 2025-09-09T15:51:37Z

Seems like a strange expectation by the Heatmap code; if I change the hv-anndata interface to basically

return values.flatten() if flat else values.T

it starts to work, I just don’t understand why the values API is expected to work like that:

holoviews/holoviews/plotting/bokeh/heatmap.py

Lines 123 to 124 in 31209ce

    
           zvals = aggregate.dimension_values(2, flat=False) 
        
           zvals = zvals.T.flatten()

When run for the anndata version, values is called 3 times for A[:, :], and the values come out in the exact same order, only that it seams like the heatmap plotting code expects the flat=False version to be transposed for some reason:

dim=A[:, :], expanded=True, flat=True
  File "…/holoviews/plotting/plot.py", line 958, in update
    return self.initialize_plot()
  File "…/holoviews/plotting/bokeh/element.py", line 2172, in initialize_plot
    ranges = self.compute_ranges(self.hmap, key, ranges)
  File "…/holoviews/plotting/plot.py", line 617, in compute_ranges
    self._compute_group_range(group, elements, ranges, framewise,
  File "…/holoviews/plotting/plot.py", line 727, in _compute_group_range
    data_range = el.range(el_dim, dimension_range=False)
  File "…/holoviews/core/data/__init__.py", line 201, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "…/holoviews/element/raster.py", line 964, in range
    return super().range(dim, data_range, dimension_range)
  File "…/holoviews/core/data/__init__.py", line 201, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "…/holoviews/core/data/__init__.py", line 529, in range
    lower, upper = self.interface.range(self, dim)
  File "…/holoviews/core/data/interface.py", line 414, in range
    column = dataset.dimension_values(dimension)
  File "…/holoviews/core/data/__init__.py", line 201, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "…/holoviews/core/data/__init__.py", line 1178, in dimension_values
    values = self.interface.values(self, dim, expanded, flat)

dim=A[:, :], expanded=True, flat=False
  File "…/holoviews/plotting/plot.py", line 958, in update
    return self.initialize_plot()
  File "…/holoviews/plotting/bokeh/element.py", line 2201, in initialize_plot
    self._init_glyphs(plot, element, ranges, source)
  File "…/holoviews/plotting/bokeh/heatmap.py", line 154, in _init_glyphs
    super()._init_glyphs(plot, element, ranges, source)
  File "…/holoviews/plotting/bokeh/element.py", line 2101, in _init_glyphs
    data, mapping, style = self.get_data(element, ranges, style)
  File "…/holoviews/plotting/bokeh/heatmap.py", line 123, in get_data
    zvals = aggregate.dimension_values(2, flat=False)
  File "…/holoviews/core/data/__init__.py", line 201, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "…/holoviews/core/data/__init__.py", line 1178, in dimension_values
    values = self.interface.values(self, dim, expanded, flat)

dim=A[:, :], expanded=True, flat=True
  File "…/holoviews/plotting/plot.py", line 958, in update
    return self.initialize_plot()
  File "…/holoviews/plotting/bokeh/element.py", line 2201, in initialize_plot
    self._init_glyphs(plot, element, ranges, source)
  File "…/holoviews/plotting/bokeh/heatmap.py", line 154, in _init_glyphs
    super()._init_glyphs(plot, element, ranges, source)
  File "…/holoviews/plotting/bokeh/element.py", line 2101, in _init_glyphs
    data, mapping, style = self.get_data(element, ranges, style)
  File "…/holoviews/plotting/bokeh/heatmap.py", line 139, in get_data
    for v in aggregate.dimension_values(vdim)]
  File "…/holoviews/core/data/__init__.py", line 201, in pipelined_fn
    result = method_fn(*args, **kwargs)
  File "…/holoviews/core/data/__init__.py", line 1178, in dimension_values
    values = self.interface.values(self, dim, expanded, flat)

hoxbro · 2025-09-11T08:59:29Z

holoviews/operation/element.py

+                code_map = defaultdict(lambda: len(code_map))  # noqa: B023
+                order = list(map(code_map.__getitem__, ddata))


Performance seems good here:

import pandas as pd import numpy as np from collections import defaultdict from string import ascii_lowercase var = [*'mtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnkslmtropqnksl'] * 100 + [*ascii_lowercase] var_np = np.asarray(var) print(len(var), len(set(var))) code_map = defaultdict(lambda: len(code_map)) # noqa: B023 order1 = list(map(code_map.__getitem__, var)) order2 = pd.Categorical(var_np, pd.unique(var_np)).codes np.testing.assert_array_equal(order1, order2)

philippjfr · 2025-09-11T10:27:29Z

Just looked into it and it seems like that the AnnDataGridInterface is simply missing code that transposes the arrays to the expected orientations. Specifically, the values method is meant to transpose the arrays to match the order of the key dimensions, i.e. if the kdims declare [obs, var] as the dimensions then the array should be returned as the exact opposite, i.e. as var x obs. That is also the case for the expanded key dimensions.

flying-sheep · 2025-09-11T12:03:54Z

Nope! That code is here: https://github.com/holoviz-topics/hv-anndata/blob/662be7ffcc077c027aabd5676d04a5f894efe41d/src/hv_anndata/interface.py#L471-L472

philippjfr · 2025-09-11T14:02:16Z

Yeah, this almost drove me insane. I don't think the conventions of the ordering and orientations expected of the flattened arrays make much sense but there was also some weird handling in the gridded interface. I've tried to resolve this in holoviz-topics/hv-anndata#89 and tried the various conditions, which now seem to work.

flying-sheep · 2025-09-11T14:46:01Z

I’ll comment there!

hoxbro added 2 commits August 22, 2025 15:32

fix: dendrogram edgecases

9971165

Convert gridded data to dataframe

42c91bc

Add test for gridded data

b105545

flying-sheep mentioned this pull request Aug 28, 2025

Scanpy gallery holoviz-topics/hv-anndata#66

Merged

Respect vdims even if it is not the main_dim

6c79b46

Raise more informative error if for adjoned layout

3e78992

Also sort non-adjoined axis

3c51111

Add warning if adjoint_dims is not part of the two first kdims

66cdf44

hoxbro force-pushed the fix_edge_cases_dendrogram branch from fbd86b1 to 66cdf44 Compare September 8, 2025 14:15

droumis added this to NIH-NCI Sep 8, 2025

droumis assigned hoxbro Sep 8, 2025

Add comment to why sorting is needed

26d566d

hoxbro force-pushed the fix_edge_cases_dendrogram branch from 6ff8e1f to 26d566d Compare September 8, 2025 15:22

hoxbro added 4 commits September 9, 2025 12:15

Add tests for edgecases in dendrogram operation

6534f99

Add test for dendrogram in layout

e0fb3a4

Merge branch 'main' into fix_edge_cases_dendrogram

862a60a

handle raise condition in backends

412a7e4

hoxbro marked this pull request as ready for review September 9, 2025 11:57

Merge branch 'main' into fix_edge_cases_dendrogram

434d07e

hoxbro commented Sep 11, 2025

View reviewed changes

philippjfr mentioned this pull request Sep 11, 2025

Fix orientations of flat coordinate and 2D value arrays holoviz-topics/hv-anndata#89

Open

hoxbro requested a review from maximlt September 11, 2025 14:51

		code_map = defaultdict(lambda: len(code_map)) # noqa: B023
		order = list(map(code_map.__getitem__, ddata))

Uh oh!

fix: dendrogram edgecases #6669

Are you sure you want to change the base?

fix: dendrogram edgecases #6669

Conversation

hoxbro commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

flying-sheep commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flying-sheep commented Aug 26, 2025

Uh oh!

flying-sheep commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flying-sheep commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoxbro commented Aug 28, 2025

Uh oh!

flying-sheep commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoxbro commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flying-sheep commented Aug 29, 2025

Uh oh!

flying-sheep commented Sep 8, 2025

Uh oh!

hoxbro commented Sep 8, 2025

Uh oh!

flying-sheep commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoxbro commented Sep 8, 2025

Uh oh!

flying-sheep commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hoxbro Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

philippjfr commented Sep 11, 2025

Uh oh!

flying-sheep commented Sep 11, 2025

Uh oh!

philippjfr commented Sep 11, 2025

Uh oh!

flying-sheep commented Sep 11, 2025

Uh oh!

Uh oh!

hoxbro commented Aug 25, 2025 •

edited

Loading

codecov bot commented Aug 25, 2025 •

edited

Loading

flying-sheep commented Aug 26, 2025 •

edited

Loading

flying-sheep commented Aug 28, 2025 •

edited

Loading

flying-sheep commented Aug 28, 2025 •

edited

Loading

flying-sheep commented Aug 29, 2025 •

edited

Loading

hoxbro commented Aug 29, 2025 •

edited

Loading

flying-sheep commented Sep 8, 2025 •

edited

Loading

flying-sheep commented Sep 9, 2025 •

edited

Loading