590 change order of tada findnearbysites to increase efficiency #592
Conversation
efficiency updates
…to-increase-efficiency
…to-increase-efficiency
R/Utilities.R
Outdated
| } | ||
|
|
||
| # remove intermediate objects | ||
| rm(grouped.no.dates, grouped.sites, org.meta.filter, org.string, meta.string) |
There was a problem hiding this comment.
grouped.sites in row 932 (starts with date.meta <- grouped.sites %>%) was removed/is no available now.
This is causing a related rm note:
Warning message:
In rm(grouped.no.dates, grouped.sites, org.meta.filter, org.string, :
object 'grouped.sites' not foundThere was a problem hiding this comment.
Thanks for catching that - I'll take a look and fix it.
|
I added a new function TADA_NearbySitesMap for users to more easily review nearby sites. Using the same Utah example, it makes a lot more sense and the groupings also look to be correct using a variety of different buffer distances. The format of this map is quite different than others as I used different colors for each group, a lighter/more transparent color to show a distance buffer radius, and used clustering so that the user would be guided to click on where a site group or groups are located on the map and zoom in. Do you all think this approach works? Or would it be better to switch back to more of the icon marker style as we have for some of the other maps? I also removed the nearby sites icons from the TADA_FlaggedSitesMap function as I think it makes sense to consider them separately from those with suspect coordinates. This is ready for review. |
|
It looks like org_hierarchy is not being carried into the TADA_FindNearbySites when it runs within TADA_FindPotentialDuplicatesMultipleOrgs WQP_Shepherdstown_raw <- TADA_DataRetrieval(
startDate = "2019-01-01",
endDate = "2024-12-31",
bBox = c(-78.2241,38.9185,-77.1041,39.7487),
applyautoclean = TRUE,
ask = FALSE)
> WQP_Shepherdstown_test <- TADA_FindPotentialDuplicatesMultipleOrgs(
+ WQP_Shepherdstown_raw ,
+ org_hierarchy = "21VASWCB"
+ )
[1] "TADA_FindNearbySites: No org_hierarchy supplied by user. Organization will not be taken into account during metadata selection." |
|
Thanks - I'll check that out as well as the test coverage failure. |
|
I added org_hierarchy to TADA_FindNearbySites when it is run as part of TADA_FindPotentialDuplicatesMulitpleOrgs |
…to-increase-efficiency
With this test data set:
test <-TADA_DataRetrieval(statecode = "IL",
startDate = "2018-06-01",
endDate = "2018-09-30",
ask = FALSE)
This update workflow which involves grouping by buffer distance first, then only fetching NHD info for sites within the buffer distance took 2.7 min.
The old workflow which fetched all NHD catchment info, grouped by nhdplusid, then looked at buffer took 18.9 min with this data set.
The same number of nearby site results are identified in each approach. I will do some additional testing to ensure that it is the same results.