Beam centre finder #169

nvaytet · 2023-03-01T10:58:38Z

This PR adds the beam_center function to find the beam center in a SANS experiment.
The prodecure to determine the precise location of the beam center is the following:

obtain an initial guess by computing the center-of-mass of the pixels, weighted by the counts on each pixel
from that initial guess, divide the panel into 4 quadrants
compute $I(Q)$ inside each quadrant and compute the residual difference between all 4 quadrants
iteratively move the centre position and repeat 2. and 3. until all 4 $I(Q)$ curves lie on top of each other

This is described in detail in a new notebook, and is used in the existing SANS notebooks.

I do not know how to add good unit tests for the beam center finder... Suggestions welcome.

…ns are uniform

…. We need to recompute wavelength after moving pixels

nvaytet · 2023-03-01T11:02:54Z

@wpotrzebowski the final $I(Q)$results have changed, because we are masking both numerator and denominator now (this was an oversight in the old code).
We should carefully go over the results to make sure I haven't messed up.

SimonHeybrock

Some comments for now, I did not look at everything in detail yet.

SimonHeybrock · 2023-03-02T09:51:44Z

src/ess/sans/beam_center.py

+from .normalization import normalize
+
+
+def _center_of_mass(data):


Type hints missing everywhere

SimonHeybrock · 2023-03-02T09:53:04Z

src/ess/sans/beam_center.py

+    summed = data.sum(list(set(data.dims) - set(data.meta['position'].dims)))
+    v = sc.values(summed.data)
+    com = sc.sum(summed.meta['position'] * v) / v.sum()
+    return com.fields.x, com.fields.y


This assumes that the beam direction is z. Should this be checked somewhere?

I tried to make it general in terms of normal to the direction of the beam. please check

SimonHeybrock · 2023-03-02T09:53:27Z

src/ess/sans/beam_center.py

+    c = ((data['top'] - ref)**2 + (data['left'] - ref)**2 +
+         (data['bottom'] - ref)**2) / ref**2
+    out = c.sum().value
+    print(xy, out)


Suggested change

print(xy, out)

?

This should go to the logger 👍

SimonHeybrock · 2023-03-02T09:59:53Z

src/ess/sans/beam_center.py

+    pi = sc.constants.pi.value
+    phi_offset = sc.scalar(pi / 4, unit='rad')
+    phi = (sc.atan2(y=data.coords['position'].fields.y,
+                    x=data.coords['position'].fields.x) +
+           phi_offset) % (2 * (pi * sc.units.rad))
+    phi_bins = sc.linspace('phi', 0, pi * 2, 5, unit='rad')
+    quadrants = ['right', 'top', 'left', 'bottom']


Not sure I follow entirely, but given the pi/4 and naming (right, top, ...), it looks like you make an X cut into quadrants? I imagine this is fine for square detector banks, but what if the bank is rectangular? Wouldn't you get very different pixel counts in the (left/right) vs (top/bottom) quadrants?

Can you make a + cut instead to avoid this?

Nice observations. I tested a + cut instead and I get nice results, so I will go with that. Thanks.

SimonHeybrock · 2023-03-02T10:02:08Z

src/ess/sans/beam_center.py

+    # Make a copy of the original data
+    data = sc.DataArray(data=sample.data)
+    coord_list = ['position', 'sample_position', 'source_position']
+    for c in coord_list:
+        data.coords[c] = sample.meta[c].copy(deep=True)


Why are you not using data=data.copy()? Is it do avoid copying wrong wavelength/Q coords from previous iterations?

data = sc.DataArray(data=sample.data) will not make a copy of the data values, is that intentional? Note that if you have event data then this will also affect event coords.

Are there no prior masks (such as TOF, or dead pixels) that need to be copied?

Is it do avoid copying wrong wavelength/Q coords from previous iterations?

Yes

will not make a copy of the data values, is that intentional?

Yes, I don't think we need a deep copy.

Note that if you have event data then this will also affect event coords.

Not sure what you mean by this will also affect event coords.

Are there no prior masks (such as TOF, or dead pixels) that need to be copied?

hmm yes, masks are missing! Thanks. I would like to change to use da.copy() and then da.coords.clear() once I can use the latest version of Scipp, but we first need to deal with the variances broadcast issue before we can do this.

Not sure what you mean by this will also affect event coords

I mean, not copying the data implies not copying the event coords (since they are "within" the data values).

SimonHeybrock · 2023-03-02T10:12:48Z

src/ess/sans/conversions.py



-def sans_elastic(gravity: bool = False) -> dict:
+def phi(position: sc.Variable) -> sc.Variable:


Should at the very least document that is for a beam direction along z. I do not know how precisely this is fulfilled in practice.

hmm nice catch, we are not using this anymore. I forgot to remove it.

There are also other places where the same assumption is made in this code.

I left this in the end and generalized it so that we don't assume the beam is along z.

SimonHeybrock · 2023-03-02T10:23:26Z

src/ess/sans/i_of_q.py

+        lu = sc.lookup(mask, mask.dim)
+        if da.coords.is_edges(mask.dim):
+            sampling = sc.midpoints(da.coords[mask.dim])
+        else:
+            sampling = da.coords[mask.dim]
+        da.masks[name] = lu[sampling]


Hmm, is considering the midpoints the correct solution? Shouldn't we mask all bins that have overlap with a masked region instead? Something like

dim = mask.dim edges = sc.lookup(mask, dim)[da.coords[dim]] da.masks[name] = edges[dim, 1:] | edges[dim, :-1]

I made a function mask_range which basically implements what you suggest, and works for both binned and dense data. Not sure if that should go somewhere more general than in the SANS submodule, as it's fully generic.

nvaytet · 2023-03-20T10:33:48Z

Few comments about functionality

We don't need BCF for every data set. As I understand this notebooks serves as an example of existing functionality rather than something that serves as a user workflow

Yes

Tof bins are no longer used (at least explicetly) - I guess it is because we don't use DataSet any more?

Correct (and we will use DataGroup in the very near future, once we address the issue of the variances being broadcast).

Pixel height and width are now incorporated into files is it the same true for sample_pos_z_offset and monitor4_pos_z_offset?

Yes

Comment to code:
In sans2d_reduction.ipynb masking is done on sample directly (sample.masks['holder_mask'] = holder_mask). Shouldn't be dg['sample']?

They are the same python object, so it should not matter. But I need to check about maybe also masking the background run...

SimonHeybrock · 2023-03-23T06:18:32Z

src/ess/sans/beam_center.py

+    Cost function for determining how close the I(Q) curves are in all four quadrants.
+    """
+    ref = data['north-east']
+    c = ((data['north-west'] - ref)**2 + (data['south-west'] - ref)**2 +
+         (data['south-east'] - ref)**2) / ref**2


This it biased and not symmetric. One would expect a result like

AB AA

to have the same cost as others such as

BA AA

but this is not the case in this implementation.

What would you suggest? Compute the mean of all 4 and use that as ref?

That could work, yes.

SimonHeybrock · 2023-03-23T06:25:01Z

src/ess/sans/beam_center.py

+    q_bins:
+        The binning in the Q dimension to be used.
+    masking_radius:
+        The radius of the circular mask to apply to the data while iterating.


Which circular mask? To mask the beam stop?

It's this mask:

I added more details in the docstring.

SimonHeybrock · 2023-03-23T06:30:32Z

src/ess/sans/common.py

+    if (dim in da.coords) and (da.coords[dim].ndim > 1):
+        raise sc.DimensionError(
+            'Cannot mask range on data with multi-dimensional coordinate. '
+            f'Found dimensions {da.coords[dim].dims} for coordinate {dim}.')


Is it worth checking if there is an existing mask of the same name? Or else we risk dropping it silently.

SimonHeybrock · 2023-03-23T06:34:05Z

docs/techniques/sans/sans-beam-center-finder.ipynb

+    "# Add X, Y coordinates\n",
+    "sample.coords['x'] = sample.coords['position'].fields.x\n",
+    "sample.coords['y'] = sample.coords['position'].fields.y\n",


Why is this necessary?

Convenience. We hist in x and y to make images of the detector panel, instead of using the instrument view which increases size bloat of the docs pages.

SimonHeybrock · 2023-03-23T06:37:27Z

docs/techniques/sans/sans-beam-center-finder.ipynb

+   "id": "afbf9a8c-4fc7-4eb3-a477-111baa047b18",
+   "metadata": {},
+   "source": [
+    "## Making 4 quadrants\n",


Is all of this repeating the code from the Python module?

Yes, it was. I was never happy with the duplication.
I now tried to do what I did in the other notebooks, which is to break up the BCF code into smaller functions, and use those inside the notebook.

SimonHeybrock · 2023-03-23T06:38:43Z

src/ess/sans/beam_center.py

+    """
+    Find the beam center of a SANS scattering pattern.
+    Description of the procedure:
+
+    #. obtain an initial guess by computing the center-of-mass of the pixels,
+       weighted by the counts on each pixel
+    #. from that initial guess, divide the panel into 4 quadrants
+    #. compute :math:`I(Q)` inside each quadrant and compute the residual difference
+       between all 4 quadrants
+    #. iteratively move the centre position and repeat 2. and 3. until all 4
+       :math:`I(Q)` curves lie on top of each other
+


We had talked about documenting all of the things that may be done wrong, i.e., all the things you learning during implementation. Shouldn't this be documented here somewhere?

As discussed, I added some notes in the docstring of the beam_center function.

SimonHeybrock · 2023-03-24T05:31:53Z

src/ess/sans/beam_center_finder.py

+    iofq = iofq_in_quadrants(xy, *args)
+    all_q = sc.concat(list(iofq.values()), dim='quadrant')
+    ref = sc.values(all_q.mean('quadrant'))
+    c = sc.abs(all_q - ref) / ref


Didn't you have squares in the old implementation?

I did, I am not sure it matters?

Does it not affect convergence speed, maybe?

It doesn't seem to, the number of iterations is the same as before.

SimonHeybrock · 2023-03-24T05:35:30Z

src/ess/sans/beam_center_finder.py

+    Notes
+    -----


👍

Maybe place under the parameter section, since users will often want to read the params, but not the notes?

… number of Q bins. remove circular mask in beam center finder

SimonHeybrock · 2023-03-28T06:22:17Z

src/ess/sans/beam_center_finder.py

+    all_q = sc.concat([sc.values(da) for da in iofq.values()], dim='quadrant')
+    ref = all_q.mean('quadrant')
+    c = (all_q - ref)**2
+    out = (sc.sum(ref * c) / sc.sum(ref)).value


You mention above that you now use a weighted mean, but I cannot really see this here. I expected that the variances would be used as weights?

I thought you meant that the signal intensity should be used as weights, to give less importance to the noisy parts with low signal?

I'm slightly reluctant to use variances as weights, as we are not computing them accurately?

nvaytet added 18 commits February 7, 2023 15:44

start adding beam center

d805f95

add masking to beam center finder to ensure all contributions to Q bi…

408ff62

…ns are uniform

add beam center finder to iofq notebook

9bb7eb2

put in logger instead of debug flag

fbb2dcc

add beam center finder to sans2d reduction notebook

70b5a01

add beam center finder notebook

27168a1

add normalization to number of counts in each quadrant

a85cd52

update notebook for iofQ reduction

eab2dfa

incorporate masks in IofQ code and second workflow notebook

4ac85bd

update notebook with procedure that uses full sans reduction

31b5966

update notebook

2d2a4a8

update beam_center code

3c9061e

start updating sans docs notebooks to use new beam center finder

6ee47c9

almost finish iofq notebook, but beam center is doing something wrong…

4a96f28

…. We need to recompute wavelength after moving pixels

update iofq notebook to use beam center finder code

99d30b7

flake8

70b3b45

update reduction notebook

c480f52

isort

d74d2ce

nvaytet requested review from SimonHeybrock and wpotrzebowski March 1, 2023 10:58

nvaytet added 2 commits March 1, 2023 12:00

remove commented code

2faaa71

Merge branch 'main' into beam_centre_finder

c20553d

nvaytet marked this pull request as ready for review March 1, 2023 11:01

nvaytet added 5 commits March 1, 2023 13:05

codespell

bf68afc

remove datagroup for now

0591808

flake8

322df4d

fix solid angle test

13ac524

update hdf5 file hashes

9071989

SimonHeybrock reviewed Mar 2, 2023

View reviewed changes

simplify using np.union1d

f8ad2ea

nvaytet and others added 3 commits March 20, 2023 14:57

raise exception in mask_range function when input has a 2d coord

cd56739

update notebooks to use new syntax for mask_range function

0c48361

Apply automatic formatting

c92e441

SimonHeybrock reviewed Mar 23, 2023

View reviewed changes

nvaytet added 9 commits March 23, 2023 09:41

add Appendix in notebook

394e725

use symmetric cost function and move appendix into function docstring

c2f3873

more details about circular mask in docstring

6f31d66

protect agains overwriting existing mask

1763178

break bcf code up and use parts inside notebook to reduce duplication

450a06f

codespell

64355c7

cleanup and add cost equation in docstring

74ef9ce

try pinning ipywidgets and ipykernel in dependencies

e9987cb

ipykernel=6.21.3 does not exist on macos

92a9035

SimonHeybrock reviewed Mar 24, 2023

View reviewed changes

move notes below parameters in docstring

c420de8

SimonHeybrock approved these changes Mar 27, 2023

View reviewed changes

nvaytet added 2 commits March 27, 2023 16:41

use weighted mean cost function, make IofQ useable with just ints for…

a51092a

… number of Q bins. remove circular mask in beam center finder

udpate other notebooks

5e5e9b6

SimonHeybrock reviewed Mar 28, 2023

View reviewed changes

nvaytet added 4 commits March 28, 2023 09:28

flake8

622f811

remove extra closing brace

87ea5d3

improve docstrings

17dbcfe

flake8

2093886

SimonHeybrock approved these changes Mar 28, 2023

View reviewed changes

wrap url inside link

11dbe38

nvaytet merged commit c8cceea into main Mar 28, 2023

nvaytet deleted the beam_centre_finder branch March 28, 2023 14:30

		from .normalization import normalize


		def _center_of_mass(data):



		def sans_elastic(gravity: bool = False) -> dict:
		def phi(position: sc.Variable) -> sc.Variable:

Uh oh!

Beam centre finder #169

Beam centre finder #169

Uh oh!

Conversation

nvaytet commented Mar 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvaytet commented Mar 1, 2023

Uh oh!

SimonHeybrock left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvaytet commented Mar 20, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

nvaytet commented Mar 1, 2023 •

edited

Loading