Inputs spec chars eh #205
Merged
Inputs spec chars eh #205
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- all dates are YYYY-MM-DD in functions, examples - changed TADAdataRetrieval inputs to "project" and "organization" to match WQP queries.
- could replace MeasureValueSpecialCharacters, more generalized, not column specific, handles % and flags numbers with commas as well - Fixed the issue flagging scientific notation as text.
- added new functions to try
ResultIdentifiers are unique to samples. There may be multiple detection limit types related to a result identifier, but result identifiers connect to one observation and one activity. This change in code streamlines the process so that duplicates are only checked on result identifiers represented in two or more rows.
This was
linked to
issues
Feb 10, 2023
Closed
cristinamullin
requested changes
Feb 13, 2023
Collaborator
cristinamullin
left a comment
There was a problem hiding this comment.
Overall these changes look great! I added a few comments/suggestions for edits & will try to dig up some examples for you with exact duplicates and potential duplicates.
| # Remove duplicate rows - turned into a test because duplicated() takes a long | ||
| # time acting on all columns in a large dataset. | ||
| if(!length(unique(.data$ResultIdentifier))==dim(.data)[1]){ | ||
| print("Duplicate records may be present. Filtering to unique records. This may take a while on large datasets.") |
Collaborator
There was a problem hiding this comment.
change to "Duplicate records are present" ?
Contributor
Author
There was a problem hiding this comment.
I wrote "may be present" because I could potentially see someone using autoclean on a dataset where they joined detection limit data to result data and thus have unique rows with the same result identifier and different detection limits. In this case, the function will check to make sure these cases are truly unique.
- updated data harmonization vignette to include information on other TADAdataRetrieval inputs (project and organization), and information on the discrepancy between WQP and dataRetrieval regarding date format. - Added more commentary on input specification origins and date format discrepancy in TADAdataRetrieval function - Added date format check to help user ensure dates are in format YYYY-MM-DD. - updated documentation for ConvertSpecialChars
cristinamullin
approved these changes
Feb 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch consists of: