Offspec data reduction #194

arm61 · 2023-08-11T11:40:16Z

Here is the OFFSPEC data reduction example.

arm61 · 2023-08-11T11:41:45Z

That final fix might break the Amor notebook (to fix it is only necessary to have a dimension='dectector_id' in the save_ort call).

docs/instruments/external/offspec/data.py

docs/instruments/external/offspec/offspec_mantid.ipynb

jl-wynen · 2023-08-11T12:11:19Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   "id": "7636a6b0-6d9e-40e6-9169-a29fa18e4b21",
+   "metadata": {},
+   "source": [
+    "## Comparison with Mantid\n",


I don't think this should be part of the docs. We want to show how to do things with Scipp and not with Mantid. That being said, the comparison is valuable.
You can leave it here for now. We'll hopefully move this to a better place.

The aim was to show that the reduction is as good as the mantid version.

I don't see this as the purpose of the documentation. We want to show people how to use it. Investigations about how well it works are separate.

jl-wynen · 2023-08-11T12:13:25Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   "outputs": [],
+   "source": [
+    "reduced_mantid_remote = client.get_dataset(reduced_mantid_up.pid)\n",
+    "reduced_mantid_local = client.download_files(reduced_mantid_remote, target=\".\")\n",


Is this really necessary? For one, it pollutes the file system with the mantid file. And also, we can use the file directly and don't have to make a roundtrip through SciCat.

What part do you mean here? Specifically the downloading it?

Specifically the downloading it?

Yes, in particular the file. But there is also no reason, IMHO, to download the dataset.

docs/instruments/external/offspec/offspec_mantid.ipynb

Co-authored-by: Jan-Lukas Wynen <[email protected]>

jl-wynen · 2023-08-11T13:23:05Z

docs/instruments/external/offspec/data.py

        principal_investigator="Joshanial F. K. Cooper",
        contact_email="[email protected]",
-        end_time=datetime.now(),
+        end_time='2020-11-26T18:19:24',


Mainly for future reference, scitacean parses this as UTC unless you specify a timezone. If this is the time at ISIS, then this is coincidentally (colonialism?) correct.

jl-wynen · 2023-08-11T13:24:00Z

docs/instruments/external/offspec/offspec_mantid.ipynb

    "reduced.input_datasets.append(direct_beam_local.pid)\n",
    "reduced.name = 'Reduced reflectometry data from QCS sample'\n",
-    "reduced.used_software = [f'scipp-{sc.__version__}']\n",
+    "reduced.used_software = ['scipp-23.01.1']\n",


Why did you revert this again?

Co-authored-by: Jan-Lukas Wynen <[email protected]>

jl-wynen · 2023-09-14T14:08:12Z

I removed the SciCat stuff and comparison with Mantid. We will handle SciCat through Sciline when migrating the workflow. And the Mantid comparison is useful but shouldn't be part of the workflow example. We have a it in the history now and can recreate it later when we have a Sciline pipeline.

nvaytet · 2023-09-15T13:55:58Z

environment.yml

  - pip:
    - orsopy==1.0.1
    - sphinx-autodoc-typehints==1.18.3
+    - scitacean==23.7.0


nvaytet · 2023-09-15T13:59:51Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+    "\n",
+    "In this example, we load some test data provided by the `offspec` package.\n",
+    "We need a sample measurement (the sample is `Air | Si(790 A) | Cu(300 A) | SiO2`) and a direct beam measurement.\n",
+    "The latter was obtained by positioning the detector directly in the beam of incident neutrons and moving the sample out of the way and gives an estimate for the ISIS pulse structure as a function of time-of-flight. "


Suggested change

"The latter was obtained by positioning the detector directly in the beam of incident neutrons and moving the sample out of the way and gives an estimate for the ISIS pulse structure as a function of time-of-flight. "

"The latter was obtained by positioning the detector directly in the beam of incident neutrons and moving the sample out of the way.\n",

"It gives an estimate for the ISIS pulse structure as a function of time-of-flight. "

nvaytet · 2023-09-15T14:00:58Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   },
+   "outputs": [],
+   "source": [
+    "sample_full = scn.load_with_mantid(offspec.data.sample(), mantid_args={\"LoadMonitors\": True})\n",


We should move away from loading with Mantid (cf #208)

nvaytet · 2023-09-15T14:02:16Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+    "    name=\"Joshanial F. K. Cooper\",\n",
+    "    affiliation=\"ISIS Neutron and Muon Source\",\n",
+    "    contact=\"[email protected]\",\n",


Maybe we should have a fake name and email for our online docs?

You are probably right for data protection reasons. But on the other hand, we need to somehow acknowledge the owner of the data. I'm not sure how to do both.

You can add a line in the notebooks (at the start or the end) saying something like "The data used in this notebook was kindly provided by Joshanial F. K. Cooper (ISIS, STFC)."?

Then we can keep the name here as well. Are you concerned about the email address? I can try removing it if ORSO lets me or use a dummy otherwise.

Yeah, it was mostly the email.

nvaytet · 2023-09-15T14:04:28Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+    "### Correcting the position of detector pixels\n",
+    "\n",
+    "The pixel positions in the sample data must be modified to account for the transformation on the detector by rotating it around the sample. \n",
+    "We can achieve this by understanding that the sample has been rotated by some amount (as discussed above) and that sample measurement has the specular peak at the same pixel as the direct beam measurement has the direct beam. \n",


as discussed above

Not sure which discussion you are refering to?

Seems to have been removed. Do we need a discussion of is this comment (minus the ref to a discussion) enough?

Not sure, I would say removing "(as discussed above)" would be ok?

nvaytet · 2023-09-15T14:22:27Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   },
+   "outputs": [],
+   "source": [
+    "sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",


Suggested change

"sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",

"sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom:].mean())\n",

nvaytet · 2023-09-15T14:22:38Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   "outputs": [],
+   "source": [
+    "sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",
+    "direct_beam_mon_wav -= sc.values(direct_beam_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",


Suggested change

"direct_beam_mon_wav -= sc.values(direct_beam_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",

"direct_beam_mon_wav -= sc.values(direct_beam_mon_wav['wavelength', 15 * sc.units.angstrom:].mean())\n",

nvaytet · 2023-09-15T14:23:42Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   },
+   "outputs": [],
+   "source": [
+    "sample_mon_wav_sum = sample_mon_wav['wavelength', 2 * sc.units.angstrom:15 * sc.units.angstrom].sum()\n",


Can we define some wav_min and wav_max to avoid repeating 15 * sc.units.angstrom many times?

nvaytet · 2023-09-15T14:25:01Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+   "source": [
+    "sample_mon_wav_sum = sample_mon_wav['wavelength', 2 * sc.units.angstrom:15 * sc.units.angstrom].sum()\n",
+    "direct_beam_mon_wav_sum = direct_beam_mon_wav['wavelength', 2 * sc.units.angstrom:15 * sc.units.angstrom].sum()\n",
+    "sample_norm = sample_wav / sc.values(sample_mon_wav_sum)\n",


Are we using sc.values because of the variance broadcast issue? If so, we should add a note here, so that we don't forget to update it to something better in the future (e.g. Simon's "upper bound" method)

nvaytet · 2023-09-15T14:25:47Z

docs/instruments/external/offspec/offspec_mantid.ipynb

+    "sample_norm_hist.coords.set_aligned('theta', False)\n",
+    "direct_beam_norm_hist = direct_beam_norm.hist(\n",
+    "    wavelength=sc.geomspace(\n",
+    "        dim=\"wavelength\", start=2, stop=14, num=100, unit=sc.units.angstrom\n",


Define edges once and store in a variable?

We could also consider putting sample and direct beam in a DataGroup?

jl-wynen and others added 18 commits February 3, 2023 15:25

Add offspec collimated reduction

6446e9c

Update notebook

2dd7c02

Using the right data now

cdfc68d

Working

1d17ad7

Cleared

8216090

Normalise by bin widths

07ff79d

Remove outputs

d2ebf31

Clean up notebook

290b59e

Fix final cell

8718610

merge

ab9a78f

Amending

8f33004

Working offspec data reduction

156f77a

Now with citation and the mantin reduced data

a143cfb

Remove old file

9cccfda

Update scicat build in files

ee0b103

Use pooch for the data

b84920a

Remove mantid data file

5bee1b6

Merge branch 'main' into offspec-collimated-arm

62c86cb

arm61 and others added 6 commits August 11, 2023 13:51

flake8 and spelling

7a3f5b5

flake 8

cf91c69

Apply automatic formatting

f44cf4a

Add scitacean

364cdf5

Newer version

d1a8e15

Apply automatic formatting

1c97cec

jl-wynen reviewed Aug 11, 2023

View reviewed changes

arm61 and others added 4 commits August 11, 2023 14:29

Update docs/instruments/external/offspec/offspec_mantid.ipynb

53f49b8

Co-authored-by: Jan-Lukas Wynen <[email protected]>

Update docs/instruments/external/offspec/offspec_mantid.ipynb

9b3d0d1

Co-authored-by: Jan-Lukas Wynen <[email protected]>

Update docs/instruments/external/offspec/offspec_mantid.ipynb

21566a4

Co-authored-by: Jan-Lukas Wynen <[email protected]>

Jan-Lukas comments

dbb1d72

flake8

29a8e56

jl-wynen reviewed Aug 11, 2023

View reviewed changes

Update src/ess/reflectometry/io.py

0899260

Co-authored-by: Jan-Lukas Wynen <[email protected]>

SimonHeybrock assigned jl-wynen Aug 23, 2023

jl-wynen added 3 commits September 14, 2023 15:13

Merge branch 'main' into offspec-collimated-arm

ecd9581

Add missing import

695899e

Clean up OFFSPEC notebook

01b338f

jl-wynen requested a review from nvaytet September 14, 2023 14:08

Pass dimension argument

2200c98

nvaytet reviewed Sep 15, 2023

View reviewed changes

jl-wynen added 12 commits September 26, 2023 09:21

Remove scitacean from env

4e6825b

Add offspec notebook to toc

dbebef8

Fix typos

4d87065

Slice correctly

490b74e

Use variables for wav_min and wav_max

990b23b

Add note on dropping variances

be94e91

Define variable for edges

6c33388

Remove unused imports

d6410e6

Use scipp-hdf5 files instead of ISIS NeXus

a420ed1

Remove outdated comment

58cd470

Rename to offspec_reduction.ipynb

e760829

Remove email address

c390c35

nvaytet approved these changes Sep 27, 2023

View reviewed changes

Merge branch 'main' into offspec-collimated-arm

0d0e9ac

jl-wynen merged commit 1925c0a into main Sep 27, 2023

jl-wynen deleted the offspec-collimated-arm branch September 27, 2023 13:03

jl-wynen mentioned this pull request Sep 27, 2023

Reflectometry collimated and divergent mode #160

Closed

	"The latter was obtained by positioning the detector directly in the beam of incident neutrons and moving the sample out of the way and gives an estimate for the ISIS pulse structure as a function of time-of-flight. "
	"The latter was obtained by positioning the detector directly in the beam of incident neutrons and moving the sample out of the way.\n",
	"It gives an estimate for the ISIS pulse structure as a function of time-of-flight. "

	"sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",
	"sample_mon_wav -= sc.values(sample_mon_wav['wavelength', 15 * sc.units.angstrom:].mean())\n",

	"direct_beam_mon_wav -= sc.values(direct_beam_mon_wav['wavelength', 15 * sc.units.angstrom].mean())\n",
	"direct_beam_mon_wav -= sc.values(direct_beam_mon_wav['wavelength', 15 * sc.units.angstrom:].mean())\n",

Uh oh!

Offspec data reduction #194

Offspec data reduction #194

Uh oh!

Conversation

arm61 commented Aug 11, 2023

Uh oh!

arm61 commented Aug 11, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jl-wynen commented Sep 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants