Updates to TADA_HarmonizeSynonyms and TADA_ConvertResultUnits #619
Conversation
updates to handle USGS results separately to prevent issues w/ speciation
fixed join bug for usgs unit ref
fix usgs data bug
bug fix test
joining bug fix for joinUSGSother
detection limit bug fix
det limit conversion bug fix
edit comments for internal functions, fix issues with checking for usgs and other results
remove int objects
fix rm issue
fix setNull internal function
more work on setNull
fixes in transform section
remove test files from .gitignore
|
I should be able to work through the check issues tomorrow, @wokenny13 and @cristinamullin. They are related to conversion of depth units, so maybe something I changed in a ref file caused a problem? I'll start there with troubleshooting. |
add missing rows to final .data
|
Do you think our target for PH should be "PH_NONE_NONE_STD UNITS" or "PH_NONE_NONE_NONE"? It looks like we are going with the latter here which works just fine. "DISSOLVED OXYGEN (DO)NONE_NONE_MG/L" looks good for that one. I updated the harmonization ref to use NONE for DO SAT as well: target is now "DISSOLVED OXYGEN SATURATION_NONE_NONE%". I also fixed some issues with the TURBILITY harmonization while reviewing. I added NU to the unit conversion table as well since it is valid in WQX. I don't see STD UNITS here, but see STD********* (rejected value), have you run into anything like that?
|
|
|
||
| # Change NONE in unit, fraction, and speciation to NA for better harmonization | ||
| # Changes NONE in fraction and speciation to NA for better harmonization | ||
| # Should this be specified in the template instead? 7/25/25 cm |
There was a problem hiding this comment.
TADA_HarmonizeSynonyms was still changing NONE back to NA for PH units. Removed this line here so it is now resolving to
"PH_NONE_NONE_NONE"
The functions still changes NONE in fraction and speciation to NA for better harmonization as a generalized solution. For pH and DO, NONE is specified in the template so that overrides the generalized solution for those two. Does that still make sense as a generalized solution for all others not specified in the harmonization template... or should these lines be removed and everything done in the template (lots more rows we would need to add...)?
There was a problem hiding this comment.
@hillarymarler this has already been merged, but can you double check the logic?
|
Note that STD********* unit has this note from the WQX team: DO NOT USE, Quantity, Standard Unitretireduse None" |
Updates to result unit conversion and synonym harmonization to address the issues described by MT.
Here are the issues they described (that are addressed in this PR):
Issue 1: Harmonization of Naming Convention for Dissolved Oxygen and pH – Sample Fraction
When applying the TADA_HarmonizeSynonyms() function, the sample fraction for dissolved oxygen (DO) is not being harmonized correctly. The harmonization reference table provided by EPATADA attempts to convert the Result Sample Fraction Text for all DO samples to “NA”. This is explained in the 'TADA.ResultSampleFractionTextAssumptions' column of the reference table which states for DO, “Assumes fraction is NA for DISSOLVED OXYGEN (DO) and DISSOLVED OXYGEN SATURATION. Harmonizing to NA.”
Now the fraction for DO harmonizes to NONE
Issue 2: Harmonization of Naming Convention for Dissolved Oxygen – Speciation
The harmonization table provided by EPATADA attempts to convert the Method Speciation Name for all DO samples to “NA”. This is explained in the ‘TADA.SpecationAssumptions’ column which states for DO, “Assumes speciation is NA for DISSOLVED OXYGEN (DO) and DISSOLVED OXYGEN SATURATION. Harmonizing to NA.”
Now the fraction for DO harmonizes to NONE
Issue 3: Harmonization of Units for pH
When pH data exist in the data set, after creating a unit conversion reference file for my data set, all units associated with pH do not show up in the reference file.
pH results now harmonize correctly as std units has been added to the reference files