GATE/ROOT STIR LOR consistency test #991

robbietuk · 2022-02-17T16:24:19Z

Adds a consistency test between GATE and STIR data that was initially described in #938 (comment). This is based upon work conducted previously by @eliseemond for a TOF test.

Simulate emission of (12) point sources in GATE with known original position. Read the output ROOT files using STIR finding the closest voxel to the original position and measure distance. If distance is greater than a given tolerance then mark event as failed_event. Ensure that the majority of events are within the tolerance.

tolerance is currently set to `1.5 * norm(grid_spacing).
50% of events must not "fail" for test to pass.

A possible alternative, test the mean(abs(distance)) < tolerance?

This PR has a lot of overlap with a similar TOF test in development NikEfth#23. I suggest this PR is merged into UCL/STIR and then UCL/STIR into the TOF branch and deal with conflicts there.

REMANING TODO

Fix READMEs
Cleanup directory. Add sub directories to contain data etc.
Remove unnecessary GATE files
Upload pre-generated ROOT files to Zenodo and add download methodology. Do this during the test? Check if ROOT data exists before test?
Investigate reasoning for large distance results. Some values are up to 10* tolerance
No need to generate the STIR_images (12 * 16MB = 168MB). They are not used in the Gate simulations and they could be generated in the test. Can modify generate_image.cxx and create a class methods to generate and save the image. The utilility can call both but this test can simply generate and keep in memory... thoughts?
Verify the point source in source_test${i}.mac matches the voxel position in generate_image${i}.par
Original position image should be generated rather than loaded from file.

robbietuk · 2022-02-17T17:13:21Z

recon_test_pack/ROOT_STIR_consistency/run_pretest_script.sh

+	generate_image generate_image${I}.par
+	cd Gate_macros
+	# Create main GATE macro files from template
+	sed -e s/SOURCENAME/test${I}/ main_D690_template.mac > main_D690_test${I}.mac


sed issues may occur on MacOS. I personally have to use gsed for this command to work. This is documented elsewhere but may lead to issues for us in this case.

it's true, but the pretest script is only run to create the files, so presumably only needs running once in a while. Let's live with it (maybe just add a TODO comment in the file)

Agreed. The hroot and root files can be downloaded. run_pretest_script.sh does not need to be run for each test.

robbietuk · 2022-02-17T18:34:41Z

Current status. All tests pass with the Gate generated data and take about 1.5 seconds each. The tests are stacked as follows.

      Start  1: test_consistency_root_1
 1/73 Test  #1: test_consistency_root_1 ..............................................   Passed    1.61 sec
      Start  2: test_consistency_root_2
 2/73 Test  #2: test_consistency_root_2 ..............................................   Passed    1.36 sec
      Start  3: test_consistency_root_3
 3/73 Test  #3: test_consistency_root_3 ..............................................   Passed    1.36 sec
      Start  4: test_consistency_root_4
 4/73 Test  #4: test_consistency_root_4 ..............................................   Passed    1.36 sec
      Start  5: test_consistency_root_5
 5/73 Test  #5: test_consistency_root_5 ..............................................   Passed    1.29 sec
      Start  6: test_consistency_root_6
 6/73 Test  #6: test_consistency_root_6 ..............................................   Passed    1.30 sec
      Start  7: test_consistency_root_7
 7/73 Test  #7: test_consistency_root_7 ..............................................   Passed    1.34 sec
      Start  8: test_consistency_root_8
 8/73 Test  #8: test_consistency_root_8 ..............................................   Passed    1.35 sec
      Start  9: test_consistency_root_9
 9/73 Test  #9: test_consistency_root_9 ..............................................   Passed    1.33 sec
      Start 10: test_consistency_root_10
10/73 Test #10: test_consistency_root_10 .............................................   Passed    1.34 sec
      Start 11: test_consistency_root_11
11/73 Test #11: test_consistency_root_11 .............................................   Passed    1.28 sec
      Start 12: test_consistency_root_12
12/73 Test #12: test_consistency_root_12 .............................................   Passed    1.29 sec

KrisThielemans

looks good. I'm a bit confused on the relation between test_view_offset_root.cxx and test_consistency_root.cxx.

Also, a question is where we put the GATE macros. etc. It feels strange to me to have them in recon_test_pack, as that is currently standalone (scripts and data, used on "installed STIR"). I note that files for recon_test_pack/run_root_GATE.sh sit in examples/GATE_test_macro (even that's incomplete). That is of course a bit confusing as well. Depends a bit how independent the test is from these files. (We could have them in examples but include them in the Zenodo upload for instance)

KrisThielemans · 2022-02-18T08:08:33Z

I'm a bit confused on the relation between test_view_offset_root.cxx and test_consistency_root.cxx.

oh, you want to merge this first, and then sort out the TOF stuff later. agreed.

KrisThielemans · 2022-02-18T08:20:32Z

They are not used in the Gate simulations and they could be generated in the test. Can modify generate_image.cxx and create a class methods to generate and save the image. The utilility can call both but this test can simply generate and keep in memory... thoughts?

Splitting generate_image such that GenerateImage ends up in stir/Shape is easy enough. @markus-jehl might have done this already?

The main weakness here is the independence of the point source locations in the GATE macros and .par files. I'd prefer to run GATE from the STIR images, but maybe that's a job for the STIR-GATE-Connection/ so too complicated for here. As you prefer.

markus-jehl · 2022-02-18T09:29:54Z

No, I haven't looked at generate_image before.

robbietuk · 2022-02-18T20:46:24Z

Two step process:

Split GenerateImage.compute() into two to add save_image() method.
Move GenerateImage to its own header and source and tie to Shape. I may well have made some mistake or not finished the documentation.

New test pass with this modification, as does generate_image 😄

src/include/stir/Shape/GenerateImage.h

KrisThielemans

some comments on generate_image. Add a line about it in release_5.0.htm (in the developer's section I guess). Can then also add a line about the test there somewhere

src/Shape_buildblock/GenerateImage.cxx

src/include/stir/Shape/GenerateImage.h

src/recon_test/test_view_offset_root.cxx

src/utilities/generate_image.cxx

robbietuk · 2022-02-27T19:33:39Z

Also, a question is where we put the GATE macros. etc. It feels strange to me to have them in recon_test_pack, as that is currently standalone (scripts and data, used on "installed STIR"). I note that files for recon_test_pack/run_root_GATE.sh sit in examples/GATE_test_macro (even that's incomplete). That is of course a bit confusing as well. Depends a bit how independent the test is from these files. (We could have them in examples but include them in the Zenodo upload for instance)

I think this is part of the final steps now. Where do these macro files belong? Secondly, how do we run the test if the data is not present?

KrisThielemans · 2022-02-27T21:24:32Z

Where do these macro files belong?

I wouldn't mind putting them in examples/GATE_point_source_test_macros. As long we have a README.md explaining things.

Secondly, how do we run the test if the data is not present?

I suggest that the download is part of the test. Two options:

a script to do this. Could re-use https://github.com/SyneRBI/SIRF-Exercises/blob/master/scripts/download_data.sh#L73-L179.
use CbMake script mode , where all the above is a one liner https://github.com/SyneRBI/SIRF/blob/6713bab5838926e07d264f757bed5f8136b76db6/src/xGadgetron/cGadgetron/tests/CMakeLists.txt#L26

The 2nd option seems far simpler, even though it relies on CMake being present at test-time (which could be a minor problem with the conda recipe, but we can fix that).

robbietuk

Suggested fixes to simplify, speed up, and reduce data size for the test. Some future (TOF) proofing too.

recon_test_pack/ROOT_STIR_consistency/Gate_macros/digitiser_D690.mac

robbietuk · 2022-03-25T18:02:07Z

recon_test_pack/ROOT_STIR_consistency/Gate_macros/geometry_D690.mac

@@ -0,0 +1,117 @@
+#	CYLINDRICAL


This D690 geometry does not match that in the SGC.

Could add a scanner without blocks/buckets and only crystals. Probably not worth it as it wouldnt tests any of the other functionality.

Gate geometry has been updated to better reflect the blocks and buckets. We could continue to modify this.

recon_test_pack/ROOT_STIR_consistency/Gate_macros/digitiser_D690.mac

recon_test_pack/ROOT_STIR_consistency/Gate_macros/geometry_D690.mac

recon_test_pack/ROOT_STIR_consistency/Gate_macros/physics.mac

For more information, please check the 2017 MIC presentation I have done during the STIR user's meeting.

…_script.sh

…-created

… with GenerateImage rather than load from file. Also fix some messaging regarding main

…try tests

…test_view_offset_root` script

robbietuk · 2022-03-28T00:32:08Z

With the latest changes to the GATE geometry and crystal material, the ROOT files are ~400kB (each so ~5MB total) with 2000-2400 coincidence events in each. These tests are all passing on my machine. Is this still too large to include on GitHub and do we still need Zenodo? Probably, and then more can be added later.

As alluded to in #938 (comment), this GATE scanner is still 90 degrees rotated so before an upload of the root files we may want to address that issue.

Also, not too happy with the verbosity of the test, as seen in the previous comment: #991 (comment). Edit: fixed in 5375973 and 8114f3c

…STIR_consistency directory

KrisThielemans

tiny clean-up comments

documentation/release_5.0.htm

examples/ROOT_files/ROOT_STIR_consistency/run_pretest_script.sh

people/eliseemond/.gitignore

KrisThielemans · 2022-03-28T21:06:38Z

do we still need Zenodo?

On the one hand, Zenodo/github often creates time-outs, but on the other, test data size can only grow as you say. I therefore still think we should do that.

I do think we have to remove the quarter stuff first (and therefore adjust the GATE scanner).

Do you have some plots to show?

robbietuk · 2022-03-28T21:13:10Z

I therefore still think we should do that.

Okay

I do think we have to remove the quarter stuff first (and therefore adjust the GATE scanner).

Agreed. This PR or a seperate with local testing for now?

Do you have some plots to show?

No plots needed I think. The output of the python script is below. Changing the GATE scanner geometry, crystal size, and crystal material appears to have alligned the tests fairly well and simplified a lot. This is with the Blocky GATE D690 scanner and STIR cylindrical PET geometry.

Less than 5% error for each point source. A failed event occurs when the closest voxel in the LOR to the original coordinates is greater than ~6.66 mm

Information regarding pass fail rate of the view offset test
 ID || The Number of Events  ||  The Number of Failed Events  ||  Percentage Error
-----------------------------------------------------------------------------------
1  ||           2404         ||              72                ||      3.0      
2  ||           2407         ||              72                ||      2.99      
3  ||           2409         ||              76                ||      3.15      
4  ||           2384         ||              73                ||      3.06      
5  ||           2074         ||              9                ||      0.434      
6  ||           2089         ||              8                ||      0.383      
7  ||           2408         ||              62                ||      2.57      
8  ||           2309         ||              60                ||      2.6      
9  ||           2406         ||              81                ||      3.37      
10  ||           2451         ||              74                ||      3.02      
11  ||           2011         ||              9                ||      0.448

No real offset in any axis. The TOTAL BIAS test is to ensure there isn't a general shift in the axis.

Mean offset in each axis for each source position
 ID ||   x   ||   y   ||   z
------------------------
1  || -2.1  || 0.61  || 0.36  
2  || 2.0  || -0.52  || 0.26  
3  || -0.54  || -2.0  || 0.33  
4  || 0.54  || 2.0  || 0.3  
5  || -0.72  || -0.3  || 0.42  
6  || -0.24  || 0.81  || 0.45  
7  || -2.0  || 0.54  || -0.41  
8  || 2.0  || -0.61  || -0.45  
9  || -0.64  || -2.1  || -0.43  
10  || 0.58  || 2.0  || -0.42  
11  || -0.66  || -0.36  || -0.65  
TOTAL BIAS
   ||   x    ||   y    ||   z
   || -0.16   || 0.011   || -0.021
Done

KrisThielemans · 2022-03-29T07:30:51Z

we could merge it now, but then the test will have to be disabled in CMakeLists.txt as it will fail due to no data presumably.

KrisThielemans · 2022-03-29T08:04:23Z

Less than 5% is great, but if some of these are 10cm off, it's still weird. I guess a histogram plot would reassure me (truncated in vertical axis as it'll be dominated by the peak).

src/recon_test/test_view_offset_root.cxx

KrisThielemans · 2022-03-29T08:00:56Z

examples/ROOT_files/ROOT_STIR_consistency/DebugScripts/debug_view_offset_consistency.py

+        print(f"{key}  ||           {num_events}         "
+              f"||              {num_failed_events}                "
+              f"||      {round_sig(percentage, 3)}      {warning_msg}")


if you happen to know how to put consistent width in printing in Python, would be good to add it here

I had way too much fun doing this with copilot so the new changes are exessive but it looks pretty 😄

Information regarding pass fail rate of the view offset test SourceID | Number of Events | Failed Events | Failure (%) --------------------------------------------------------------------------- 1 | 2404 | 72 | 3.0 2 | 2407 | 72 | 2.99 3 | 2409 | 76 | 3.15 4 | 2384 | 73 | 3.06 5 | 2074 | 9 | 0.434 6 | 2089 | 8 | 0.383 7 | 2408 | 62 | 2.57 8 | 2309 | 60 | 2.6 9 | 2406 | 81 | 3.37 10 | 2451 | 74 | 3.02 11 | 2011 | 9 | 0.448 Mean offset in each axis for each source position SourceID | Mean Offset (x) | Mean Offset (y) | Mean Offset (z) ----------------------------------------------------------------------- 1 | -2.05 | 0.61 | 0.357 2 | 1.97 | -0.519 | 0.264 3 | -0.542 | -2.03 | 0.33 4 | 0.536 | 2.01 | 0.304 5 | -0.724 | -0.299 | 0.425 6 | -0.242 | 0.814 | 0.453 7 | -1.97 | 0.542 | -0.412 8 | 1.99 | -0.606 | -0.449 9 | -0.641 | -2.06 | -0.432 10 | 0.584 | 2.02 | -0.419 11 | -0.659 | -0.358 | -0.647 TOTAL BIAS IN EACH AXIS Total Bias (x) | Total Bias (y) | Total Bias (z) -------------------------------------------------- -0.159 | 0.0114 | -0.0206 Done

examples/ROOT_files/ROOT_STIR_consistency/DebugScripts/debug_view_offset_consistency.py

…rance. Other small fixes

robbietuk · 2022-03-29T16:31:47Z

Histogram plot of ALL offsets from origin for point source 9, which has the largest proportion of failed events.

robbietuk · 2022-03-29T16:39:16Z

we could merge it now, but then the test will have to be disabled in CMakeLists.txt as it will fail due to no data presumably.

I have disabled the test. Lets merge it now (if ready) and this test can be used on seperate branch to try to fix #938

robbietuk force-pushed the ROOT/ViewOffsetTest branch from 09191f9 to 797affe Compare February 17, 2022 16:29

robbietuk commented Feb 17, 2022

View reviewed changes

KrisThielemans reviewed Feb 18, 2022

View reviewed changes

robbietuk force-pushed the ROOT/ViewOffsetTest branch from a843b44 to 3531891 Compare February 18, 2022 20:45

robbietuk commented Feb 18, 2022

View reviewed changes

src/include/stir/Shape/GenerateImage.h Outdated Show resolved Hide resolved

KrisThielemans requested changes Feb 19, 2022

View reviewed changes

robbietuk commented Mar 25, 2022

View reviewed changes

eliseemond and others added 16 commits March 25, 2022 14:45

Some changes for positron range: one medium

1e101d0

Files for STIR+ROOT check (symmetries, unlisting etc.)

17760f1

The .hroot files were missing

275d63d

Gate/STIR time resolution calibration

63e3ea1

For more information, please check the 2017 MIC presentation I have done during the STIR user's meeting.

Update README.md

d9ef81f

README.md for ROOT consistency check for TOF

353c286

[ci skip] Create STIR ROOT consistency test and supporting files

4486c70

[ci skip] Remove Gate main_D690_test* files as created by run_pretest…

4664e15

…_script.sh

[ci skip] Add generation of .hroot files from template and remove pre…

e208879

…-created

[ci skip] Remove lm_to_projdata files. Not needed.

7c60e7b

[ci skip] Sed and CMakeList test fixes

0e21a64

Changes to generate image to split saving from compute()

ab3c83c

[ci skip] Split generate_image (utility) and GenerateImage (class)

05574ef

[ci skip] Update root consistency test to generate point source image…

601681c

… with GenerateImage rather than load from file. Also fix some messaging regarding main

[ci skip] Rename test_lm_data_closest_approach to process_list_data

e236972

[ci skip] Documentation

224a6a8

robbietuk mentioned this pull request Mar 27, 2022

ROOT view/detector offset #938

Closed

robbietuk added 5 commits March 27, 2022 14:05

Merge remote-tracking branch 'origin/master' into ROOT/ViewOffsetTest

f153aae

[ci ski[] Changes to digitiser and physics model to improve the geome…

00ddb6e

…try tests

[ci skip] Use an improved D690 scanner geometry

b3e00b7

[ci skip] Add a debug script for interpreting the results output of `…

b99f443

…test_view_offset_root` script

[ci skip] Geometry changes and remove D690 from GATE mac filenames

66140f0

robbietuk added 2 commits March 27, 2022 19:13

[ci skip] Run all 12 point source tests in same executable from ROOT_…

5375973

…STIR_consistency directory

[ci skip] Run only one test_view_offset_root test

8114f3c

KrisThielemans requested changes Mar 28, 2022

View reviewed changes

documentation/release_5.0.htm Outdated Show resolved Hide resolved

examples/ROOT_files/ROOT_STIR_consistency/run_pretest_script.sh Outdated Show resolved Hide resolved

people/eliseemond/.gitignore Outdated Show resolved Hide resolved

[ci skip] cleanup and simplifications

00820a8

KrisThielemans requested changes Mar 29, 2022

View reviewed changes

robbietuk added 2 commits March 29, 2022 09:18

[ci skip] Improve table drawing by using padding for entry lengths

4040c15

[ci skip] cmake test fails if more than 5% of events are outside tole…

e5b68f5

…rance. Other small fixes

robbietuk added 2 commits March 29, 2022 09:36

[ci skip] Debug script improvements and comments

1647acc

Disable test_view_offset_root test as data not uploaded

ef8b658

KrisThielemans added this to the v5.0.1 milestone Mar 29, 2022

KrisThielemans merged commit 07a4b68 into UCL:master Mar 29, 2022

robbietuk mentioned this pull request Mar 30, 2022

Root/scanner rotation changes #1016

Merged

8 tasks

robbietuk deleted the ROOT/ViewOffsetTest branch April 6, 2022 00:31

This was referenced Apr 9, 2022

Improve tests for ROOT consistency NikEfth/STIR#30

Merged

STIR configuration is slow because of downloading test_view_offset_root data #1022

Closed

KrisThielemans mentioned this pull request Dec 15, 2022

upgrade parallelproj requirement to 1.2 #1033

Closed

KrisThielemans mentioned this pull request Sep 20, 2023

run_root_GATE.sh should do some tests on actual coincidence decoding #751

Closed

Uh oh!

GATE/ROOT STIR LOR consistency test #991

GATE/ROOT STIR LOR consistency test #991

Uh oh!

Conversation

robbietuk commented Feb 17, 2022 • edited by KrisThielemans Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

REMANING TODO

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robbietuk commented Feb 17, 2022

Uh oh!

KrisThielemans left a comment

Choose a reason for hiding this comment

Uh oh!

KrisThielemans commented Feb 18, 2022

Uh oh!

KrisThielemans commented Feb 18, 2022

Uh oh!

markus-jehl commented Feb 18, 2022

Uh oh!

robbietuk commented Feb 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

KrisThielemans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robbietuk commented Feb 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KrisThielemans commented Feb 27, 2022

Uh oh!

robbietuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robbietuk commented Mar 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KrisThielemans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KrisThielemans commented Mar 28, 2022

Uh oh!

robbietuk commented Mar 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KrisThielemans commented Mar 29, 2022

Uh oh!

KrisThielemans commented Mar 29, 2022

robbietuk commented Feb 17, 2022 •

edited by KrisThielemans

Loading

robbietuk commented Feb 18, 2022 •

edited

Loading

robbietuk commented Feb 27, 2022 •

edited

Loading

robbietuk commented Mar 28, 2022 •

edited

Loading

robbietuk commented Mar 28, 2022 •

edited

Loading

robbietuk commented Mar 29, 2022 •

edited

Loading