All committers have signed the CLA.

It might be worth looking at how this handles the tests in data.table and knitr. Both of them do not use testthat or RUnit.

Thanks @jimhester! I've been trying to dig up good example packages and will definitely try these out.

Regarding the CI checks, I'm able to successfully run tests on my machine (macOS R 4.0.3). I tried testing on an older R install (3.6.2) as well but ran into curl dynamic library issues - I think these are just issues with my machine, but can explore further if that might be helpful. At least for the linux builds, the issue seems to be with dependency installs.

Do you have any suggested actions I should take to get builds passing?

I think I've added some reasonably generalized fallbacks for trying to collect the test traces. In trying to test an RUnit package (yaml) and a bespoke testing package (data.table), I encountered a couple other ways that srcrefs might be captured.

Just to document the process a bit:

testthat

Packages tested with testthat are evaluated with the ./tests directory as the working directory. If any srcrefs are from a relative filepath, we assume we're using testthat and any relative srcref filepaths can be assumed to be srcrefs of tests.

Example captured trace (using package, praise)

> cov <- package_coverage("praise")
> attr(cov, "tests")[1]
$`./test.R:17:3:17:32:3:32:17:17`
$`./test.R:17:3:17:32:3:32:17:17`[[1]]
test_that("template without praise word", {
  str <- "This is just a random string"
  expect_equal(praise(str), str)
  expect_equal(praise(""), "")
})

$`./test.R:17:3:17:32:3:32:17:17`[[2]]
expect_equal(praise(str), str)

$`./test.R:17:3:17:32:3:32:17:17`[[3]]
praise(str)

RUnit

If no srcrefs in the call stack are relative paths, we check to see if any of the srcrefs that are within the temporary directory that covr installs the package into (.libPaths()[[1]] during coverage execution).

Example captured trace (using package, yaml)

> cov <- package_coverage("r-yaml")
> attr(cov, "tests")[1]
$`/private/var/folders/123abc/T/Rtmp123/R_LIBS123/yaml/tests/test_as_yaml.R:121:3:121:91:3:91:121:121`
$`/private/var/folders/123abc/T/Rtmp123/R_LIBS123/yaml/tests/test_as_yaml.R:121:3:121:91:3:91:121:121`[[1]]
result <- as.yaml(list(foo=list(bar=list('foo', 'bar'))), indent.mapping.sequence = TRUE)

Other

Finally, if all else fails, just save the whole call stack up until the call to covr:::count.

Example captured trace (using package, data.table)

> cov <- package_coverage("data.table")
> attr(cov, "tests")[2]
[[1]]
[[1]][[1]]
data.table(a = 1:2)

For testing data.table, I removed all but one test script. Otherwise it takes quite a while to run.

I think that these address a reasonable span of cases, but if there are others that I should look into I'd be happy to address them.

@jimhester, I see that CLAassistant picked up a couple different emails in my commits. I would prefer to have all the commits under one email. Would you mind if I reintroduced a new PR after squashing the changes and re-committing under only one email?

You shouldn't need to open a new PR, you should be able to change the email on the commits with interactive rebase.

Alternatively if you just add the other email to your list of emails on GitHub that would also solve the issue I think.

Packages tested with testthat are evaluated with the ./tests

This is not quite correct. R CMD check (and covr) executes all packages tests with the working directory in the tests directory. testthat further changes the working directory to tests/testthat when running its tests.

Thanks @jimhester - will update this in the description.

@jimhester - I went ahead and just removed the working directory behavior. Now, both testthat and RUnit traces are picked up by filtering for calls whose source files are within the temporary library (the second method documented above). The relative directory as a way of subsetting the call stack felt like a hack at the start, so I'm happy to replace it with something more concrete.

I edited my previous commits to all be from the same email, and the CLA looks good now. (Just a note, the CLA mentions the project is "current located at https://github.com/jimhester/covr or its successors or assigns" - wanted to raise that to your attention in case it's an oversight in the r-lib migration that needs to be updated).

Otherwise, I think this is in a good place unless you have other behavioral concerns. I looked into the failing CI builds, but I think these failures are holdovers from current master db5423f.

As far as I can tell, the builds pass with the exception of an issue that also currently exists in master - a dependency on digest that is unavailable in 3.2.

@jimhester - is there anything more that you'd like me to look into before considering this PR?

Thanks for working on this, I think it looks good as is.

This seems to break tests: git bisect identified this pull request as a source of test failures that occur in a very similar way on the main branch. Checks seem to have passed for this PR, was this a change in the testing environment (base R, testthat, ...)?

I'm seeing:

Failure (test-record_tests.R:13:3): covr.record_tests causes test traces to be recorded
length(attr(cov_func, "tests")) is not strictly more than 0L. Difference: 0

Failure (test-record_tests.R:14:3): covr.record_tests causes test traces to be recorded
length(attr(cov_func, "tests")[[1]]) is not strictly more than 0L. Difference: 0

Failure (test-record_tests.R:19:3): covr.record_tests records test indices and depth for each trace
ncol(cov_func[[1]]$tests) not equal to 2L.
target is NULL, current is numeric

Failure (test-record_tests.R:20:3): covr.record_tests records test indices and depth for each trace
colnames(cov_func[[1]]$tests) not equal to c("test", "depth").
target is NULL, current is character

Error (test-record_tests.R:25:3): covr.record_tests test traces list uses srcref key names
Error in `expect_match(names(attr(cov_func, "tests")), "\\w+(:\\d+){4,8}")`: is.character(act$val) is not TRUE
Backtrace:
 1. testthat::expect_match(names(attr(cov_func, "tests")), "\\w+(:\\d+){4,8}") test-record_tests.R:25:2
 2. base::stopifnot(is.character(act$val)) /Users/kirill/git/R/testthat/R/expectations-matches.R:35:2

This occurs an M1 Mac.

Thanks @krlmlr. I don't have a physical M1 machine to test against, but it looks like there are a few web hosted offerings that I might be able to look into to do some investigation. I'll try my best to reproduce a similar setup to investigate where things are going wrong.

Sorry for a bit of a delay - it took me a while to hunt down an M1 system to test against.

Initially, I was able to see the exact same tests failing, but after rebuilding and reinstalling they seem to disappear. I was only able to get these exact test failures if I installed the version of covr from CRAN before running the test suite. @krlmlr, could you rerun the test suite with a fresh install from source and see if the tests still fail?