Wow @jojoelfe this is so cool!! I love it, and its great to see you dive in with the PR. I think this will be functionality that a lot of people find very useful and that we definitely want to support.

I am not sure if you would want to pull this in its current form, but I though it might be interesting as a prototype. I understand that there a underlying changes coming that will allow for a cleaner implementation.

As you note there is a PR coming soon #885 that should make this implementation easier (though I think @jni and @GenevieveBuckley were going to hold off on rotation / general affine transforms for now, though maybe we will want to think about that a bit more now in light of this PR. We could still get #885 in first with just translate and scale and then make a new PR for rotation/affine right after).

I think I'd recommend we hold off on further work on this PR until those PRs go in, because then as you say

The main logic is in the image layer, while it probably should be in the base layer. However, I could not figure out how to introduce modes/UI modifications in the base layer.

this should become much easier.

It would be neat to add a symbol, like a pin, to the fixed point but I am not sure how to do this in an image layer.

I'd actually like us to reuse the same bounding box / highlight functionality that we have in the shapes layer for rotating / resizing boxes. As this resizing will now be possible for every layer the highlighting code will move out of the shapes layer and onto the base layer. There is some discussion of this in #772. This could come in a separate PR though after this PR is merged.

It only works for 2D data. I don't see a big reason why this should not easily scale to 3D, but I haven't thought about it much.

I think once we have #885 in which, will allow for nD transforms / coordinate systems it will become easier to think about how to extend this to 3D.

Does this approach make sense to you?

Yes, waiting for #885 and #772 would make things a lot cleaner.

I think together with #851 and #970 this will be a super powerful way to keep multi-modality imaging data organized.

I would love to help in any way so let me know if there is anything I can do.

I would love to help in any way so let me know if there is anything I can do.

I wonder if we should make an InteractionBox object that lives in napari/layers/interaction_box.py and has a simple model consisting of the edge / corner / center / rotation handles vertices. This object would also have the mouse functions for resizing. We could pull a lot of this code out of the shapes layer, which would hopefully make that easier to maintain. We could then have a corresponding visual that showed to lines and vertices of the highlight in the highlight color. We could then have one of the objects be attached to the base layer and it could be used to either resize selected objects like in the shapes layer right now or when resizing whole layers.

How does this sound @shanaxel42 @jni @kne42 @jojoelfe?

If the others agree this is a good approach do you want to take a stab at this @jojoelfe in a new PR which would only remove the interaction box from shapes, put it on the base layer while preserving the functionality currently in shapes (and hopefully reducing some of the complexity in shapes) while preparing to support the functionality in this PR in followup PRs. I can provide more guidance / detail if needed.

Note this is the interaction box I am referring too which right now lives inside shapes but really is more generally useful:

@sofroniewn

• I think rotations are pretty far off, unfortunately. The main issue is that we are n-dimensional, and if you rotate in the 2D plane then roll the axes, now you are doing arbitrary reslicing and interpolation. We don't even have a concept of that right now anywhere in the code base. It would be feasible to restrict rotations to 2D planes and e.g. reset/ignore them when rolling the axes, but that seems icky, to say the least. I'm not sure it's desirable.

• other than that, I do think the interaction box should be reused for this kind of functionality, that makes perfect sense.

@jni
Would it be possible to allow rotations after slicing? For example the base layer could expose a slice_transform property, that allows a 2D or 3D slice to be transformed before painting on the canvas. I can't really judge how confusing this would be if this behaves different than scale/translate and whether it is to icky in the long-term

@sofroniewn
I will give the InteractionBox object a shot in a separate pull request.

I will give the InteractionBox object a shot in a separate pull request.

Great, thanks for claiming that!! I want to give you a heads up that we're planning to refactor some of the mouse function code - see #694, which I think can wait till after you're done, but I wanted to put it on your radar. Also don't hesitate to start a WIP PR and reach out early if you hit stumbling blocks - maybe @shanaxel42 and I jump into help

Would it be possible to allow rotations after slicing?

It's a possibility. I want to give all this some serious thought, which personally I havn't done yet.

I think rotations are pretty far off, unfortunately. The main issue is that we are n-dimensional, and if you rotate in the 2D plane then roll the axes, now you are doing arbitrary reslicing and interpolation. We don't even have a concept of that right now anywhere in the code base. It would be feasible to restrict rotations to 2D planes and e.g. reset/ignore them when rolling the axes, but that seems icky, to say the least. I'm not sure it's desirable.

I think we need to find an acceptable compromise, rotations are really important to have (even if only for 2D planes).

Is it silent reset/ignore behaviour that gives you icky feelings? I personally would feel ok with a pop-up warning that says something like: "Hey you are trying to roll the axes but your image has been rotated in 2D, so this is not supported. Do you want to undo the rotation and continue rolling the axes, or do you want to cancel?"

Additionally, we could also support image flips in nd, and therefore get rotations that are multiples of 90 degrees and won't require arbitrary reslicing. I'm sure somebody wants this as a feature, even if it doesn't solve the specific use case presented here.

Speaking as someone who needs support for arbitrary reslicing (which should probably happen as a corollary of #885 ), I'd be happy seeing this PR blocked until napari can do it naturally. I don't think implementing partial support for this feature by writing code for user warnings and undo operations is a good plan, given that a lot of that code would be totally obviated once arbitrary reslicing is around.

Speaking as someone who needs support for arbitrary reslicing (which should probably happen as a corollary of #885 )

Just FYI, arbitrary reslicing is not a direct corollary of #885, as Nick says there needs to be another PR for rotation/affine transformations before arbitrary slicing can exist.

Just FYI, arbitrary reslicing is not a direct corollary of #885, as Nick says there needs to be another PR for rotation/affine transformations before arbitrary slicing can exist.

Yeah, maybe corollary is too strong of a word :) I should have said, I really hope the world coordinates implementation makes it super easy to implement arbitrary reslicing (by allowing transformations of the clipping plane... Edit: or by allowing transformations of the data's coordinate space, which is what this PR would require)

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/napari-interactive-3d-image-transformations/44620/2

This will be superseeded by #3577