Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Updated QAPP functions and created censored data function placeholder #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jakegreif merged 2 commits into from
Mar 22, 2022
Merged

Updated QAPP functions and created censored data function placeholder #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
807b11d
Adding QAPP functions and adding placeholder for censored data functions
cristinamullin Mar 22, 2022
379eba7
Merge branch 'develop' into filtering-censored-QAPP
cristinamullin Mar 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions R/CensoredData.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#Censored Data

##Two objectives:
#TransformCensoredData
#GenerateCensoredDataStats

127 changes: 57 additions & 70 deletions R/ResultFlagsIndependent.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,7 @@ AboveNationalWQXUpperThreshold <- function(.data, clean = FALSE){
}



#' Check Result Value Against WQX Lower Threshold
#'
#' EPA's Water Quality Exchange (WQX) has generated statistics and data from
Expand Down Expand Up @@ -488,104 +489,90 @@ BelowNationalWQXUpperThreshold <- function(.data, clean = FALSE){
}



#' Flag or remove result without an approved QAPP
#'
#' Organizations that submit data to the Water Quality Portal (WQP) sometimes
#' include metadata with their results to indicate if the data produced has an
#' approved Quality Assurance Project Plan (QAPP). This check reviews data
#' submitted under the column "QAPPApprovedIndicator". When clean=false,
#' a column will be appended to flag results that do not have an approved QAPP.
#' When clean = TRUE, rows with values that do not have an approved QAPP are
#' removed from the dataset and no column will be appended.This function should
#' only be used to remove data if an approved QAPP is required to use data in
#' water quality assessments
#' This check reviews data submitted under the column "QAPPApprovedIndicator".
#' Some organizations submit data for this field to indicate if the data
#' produced has an approved Quality Assurance Project Plan (QAPP) or not.
#' Y indicates yes, N indicates no. This function
#' has two default inputs: clean = TRUE, cleanNA = FALSE. The default will
#' remove rows of data where the QAPPApprovedIndicator equals "N". Users could
#' alternatively remove both N's and NA's using the inputs:
#' clean = TRUE, cleanNA = TRUE. If both clean = FALSE and cleanNA = FALSE,
#' the function will not make any changes to the dataset.
#'
#' Note: This is not a required field, so it is often left blank (NA) even if
#' the data has an associated QAPP. All states and tribes that collect
#' monitoring data using 106 funding (almost all state and tribal data in WQX)
#' are required to have an EPA approved QAPP to receive 106 funding. Therefore,
#' most of these organizations data has an approved QAPP even if the data
#' submitted to WQP is NA.
#'
#' @param .data TADA dataset
#' @param clean Boolean argument; removes data without an Approved QAPP from
#' the dataset when clean = TRUE. Default is clean = FALSE.
#' @param clean Two boolean arguments: 1) remove data where the
#' QAPPApprovedIndicator = "N" when clean = TRUE; 2) remove data where the
#' QAPPApprovedIndicator = NA when cleanNA = TRUE. Default is clean = TRUE and
#' cleanNA = FALSE.
#'
#' @return When clean = FALSE, a column is appended to the input data set that
#' flags rows without an Approved QAPP. When clean = TRUE, data without an
#' Approved QAPP is removed from the dataset.
#' @return When clean = FALSE and cleanNA = FALSE, no data is removed from the
#' dataset.
#'
#' @export
#'

QAPPapproved <- function(.data, clean = FALSE){

QAPPapproved <- function(.data, clean = TRUE, cleanNA = FALSE) {
# check that .data object is compatible with TADA
# check .data is of class data.frame
if(("data.frame" %in% class(.data)) == FALSE) {
stop("Input object must be of class 'data.frame'")
}
# check .data has required columns
if(all(c("QAPPApprovedIndicator") %in% colnames(.data)) == FALSE) {
stop("The dataframe does not contain the required fields to use TADA.
Use either the full physical/chemical profile downloaded from WQP or
download the TADA profile template available on the EPA TADA webpage.")
stop("The dataframe does not contain the required fields to use TADA. Use either the full physical/chemical profile downloaded from WQP or download the TADA profile template available on the EPA TADA webpage.")
}
# execute function after checks are passed
if(all(c("QAPPApprovedIndicator") %in% colnames(.data)) == TRUE) {

# flag data where QAPP was not approved
# make QAPPapproved.data data frame
QAPPapproved.data <- dplyr::filter(.data,
QAPPApprovedIndicator == "N")

# if there is data without an approved QAPP in the data set
if(nrow(QAPPapproved.data) != 0){
# append flag column
QAPPapproved.data$QAPPapproved <- "N"
# join QAPPapproved.data to flag.data
flag.data <- merge(.data, QAPPapproved.data, all.x = TRUE)

# flagged output
if(clean == FALSE) {
return(flag.data)
}

# clean output
if(clean == TRUE) {
# remove data without an approved QAPP
clean.data <- dplyr::filter(flag.data, !(QAPPapproved %in% "N"))
if (clean == TRUE) {
.data = dplyr::filter(.data, is.na(QAPPApprovedIndicator)==TRUE | QAPPApprovedIndicator=="Y")

if(nrow(.data)==0){
warning("All QAPPApprovedIndicator data is N")
}
}
if (cleanNA == TRUE){
.data = dplyr::filter(.data, is.na(QAPPApprovedIndicator)==FALSE)

# remove QAPPapproved column
clean.data <- dplyr::select(clean.data, -QAPPapproved)

return(clean.data)
} else {
stop("clean argument must be Boolean (TRUE or FALSE)")
}
}

# if no QAPP approved indicator data is in the data set
if(nrow(QAPPapproved.data) == 0){
warning("The dataset does not contain QAPP approval indicator data.")

return(.data)
}
if(nrow(.data)==0 & clean==TRUE){
warning("All QAPPApprovedIndicator data is NA or N")
} else if (nrow(.data)==0 & clean==FALSE){
warning("All QAPPApprovedIndicator data is NA")
}
}
return(.data)
}
}


#' Flag or remove result without an approved QAPP


#' Flag or remove results without an approved QAPP document URL provided
#'
#' Organizations that submit data to the Water Quality Portal (WQP) sometimes
#' include metadata with their results to indicate if the data produced has an
#' associated Quality Assurance Project Plan (QAPP). This check reviews data
#' submitted under the column "ProjectFileUrl" to determine if
#' a QAPP document is available to review. When clean=false,
#' a column will be appended to flag results that do not have an associated
#' QAPP document url provided. When clean = TRUE, rows with values that do not
#' This check reviews data submitted under the "ProjectFileUrl" column
#' to determine if a QAPP document is available to review. When clean=FALSE,
#' a column will be appended to flag results that have an associated
#' QAPP document url provided. When clean = TRUE, rows that do not
#' have an associated QAPP document are removed from the dataset and no column
#' will be appended. This function should only be used to remove data if an
#' accompanying QAPP document is required to use data in assessments
#' accompanying QAPP document is required to use data in assessments.
#'
#' @param .data TADA dataset
#' @param clean Boolean argument; removes data without an associated QAPP
#' document from the dataset when clean = TRUE. Default is clean = FALSE.
#'
#' @return When clean = FALSE, a column is appended to the input data set that
#' flags rows without an associated QAPP document. When clean = TRUE,
#' flags rows with an associated QAPP document. When clean = TRUE,
#' data without an associated QAPP document is removed from the dataset.
#'
#' @export
Expand All @@ -606,14 +593,14 @@ QAPPDocAvailable <- function(.data, clean = FALSE){
# execute function after checks are passed
if(all(c("ProjectFileUrl") %in% colnames(.data)) == TRUE) {

# flag data where QAPP document url is not provided
# flag data where QAPP document url is provided
# make QAPPdoc.data data frame
QAPPdoc.data <- dplyr::filter(.data, ProjectFileUrl == c("null", "NA"))
QAPPdoc.data <- dplyr::filter(.data, grepl('/', ProjectFileUrl))

# if there is data without an associated QAPP url in the data set
if(nrow(QAPPdoc.data) != 0){
# append flag column
QAPPdoc.data$QAPPDocAvailable <- "N"
QAPPdoc.data$QAPPDocAvailable <- "Y_ProjectFileUrlProvided"
# join QAPPdoc.data to flag.data
flag.data <- merge(.data, QAPPdoc.data, all.x = TRUE)

Expand All @@ -625,7 +612,7 @@ QAPPDocAvailable <- function(.data, clean = FALSE){
# clean output
if(clean == TRUE) {
# remove data without an associated QAPP url
clean.data <- dplyr::filter(flag.data, !(QAPPDocAvailable %in% "N"))
clean.data <- dplyr::filter(flag.data, grepl('/', ProjectFileUrl))

# remove QAPPDocAvailable column
clean.data <- dplyr::select(clean.data, -QAPPDocAvailable)
Expand All @@ -643,4 +630,4 @@ QAPPDocAvailable <- function(.data, clean = FALSE){
return(.data)
}
}
}
}