[ENH] Add check_data mode to deal with running with/without nan values#182
Merged
TomDonoghue merged 5 commits intomasterfrom Aug 9, 2020
Merged
[ENH] Add check_data mode to deal with running with/without nan values#182TomDonoghue merged 5 commits intomasterfrom
check_data mode to deal with running with/without nan values#182TomDonoghue merged 5 commits intomasterfrom
Conversation
check_data mode to deal with running with/without nan valuescheck_data mode to deal with running with/without nan values
Contributor
ryanhammonds
left a comment
There was a problem hiding this comment.
Hi @TomDonoghue, I went through this and left a comment about a change.
Do you think it could be useful to implement a ignore_nans option to drop nan values from the spectrum to allow fitting to exit successfully?
Member
Author
|
Just for tracking: I got an update from @mwprestonjr that this does seem to address his use case - so I think we should be all good here. Merging in. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Responds to #175
This PR adds the
check_datamode to FOOOF objects, which controls if added data is checked for NaN/Inf data-points, and fails out if so. By default,check_datais turned on, in which case objects act the same as before.However, this mode can be turned off, in which case the object plays nice with having NaN/Inf data. Model fitting will cleanly fail (no error) and so fitting can continue. This allows for fitting groups of spectra in which some spectra are NaN, and makes it easier to deal with messy shaped data in which some spectra may be missing, etc.
So far the decision here is to not change default behaviour, but add an option for users to bypass some of the data checks. This, I think, is a safer option - the user has to choose when to ignore NaN data, and should only do so if they know why it is happening, etc.
Note: this is an issue brought up by @mwprestonjr. MJ - whenever you get a chance, can you have a look at this proposal, and see if it addresses your use case, and check in whether you think this is a good solution for your purposes? Thanks!
Examples
By default, this code will error, as before:
But, with explicit toggle of the check data mode, this will no longer throw an explicit error:
This is probably most relevant for cases, with FOOOFGroup, whereby some spectra may be null:
Note, in the above, that by default (or if
set_check_data_mode(True)is called, which returns the object to default mode of having check_data on), the above code fails.