@nanosoldier runtests(["LinearAlgebra", "Pkg"])

The package evaluation job you requested has completed - no new issues were detected.
The full report is available.

Hmmm, this doesn't look correct.

Pkg is a standard library in this Julia build.


################################################################################
# Loading
#

Loading Pkg...

Loading completed after 2.08s

PkgEval succeeded after 9.83s

It looks like it only tests that it can load Pkg. But we want to actually run the Pkg test suite.

The logs show the line # Loading.

Based on https://github.com/JuliaCI/PkgEval.jl/blob/master/scripts/evaluate.jl, I think we are hitting https://github.com/JuliaCI/PkgEval.jl/blob/524847dc78c801a51a16ae7e747ebe19f8fc30a4/scripts/evaluate.jl#L184-L185

But this means that config.goal === :load is true.

So, why is config.goal getting set to :load???

I suppose I should write a sentence or two on why I want this to work.

Now that we have merged JuliaCI/julia-buildkite#483, external stdlibs will not be tested on PRs to Julia master. So I'd like some way to manually trigger external stdlib tests on a PR. And PkgEval seems like the easiest mechanism to use.

@nanosoldier runtests(["LinearAlgebra", "Pkg"], configuration = (goal=:test,), vs_configuration = (goal=:test,))

Your job submission was not accepted. Consult the server logs for more details (cc @maleadt).

@nanosoldier runtests(["DataFrames", "CSV"])

It looks like it only tests that it can load Pkg.

This is frustrating, because I cannot reproduce when I run PkgEval locally.

I tested locally with PkgEval#master¹. Here is my script I ran:

using PkgEval

config = Configuration()

packages = [
    Package(; name="LinearAlgebra"),
    Package(; name="Pkg"),
]

df = evaluate([config], packages)

When I run locally, PkgEval runs the full test suite for the stdlibs that I tried (Pkg and LinearAlgebra).

The package evaluation job you requested has completed - no new issues were detected.
The full report is available.

Okay, so this is weird - the problem is happening with CSV.jl too. For CSV.jl (log), it only loaded the package - it didn't run the tests. Why is that happening?

I wonder if the external stdlibs are getting blacklisted (at least the ones I've tried, Pkg and LinearAlgebra). And perhaps CSV.jl is blacklisted too.

Let me try running every stdlib. Surely not all of them have been blacklisted.

@nanosoldier runtests(["ArgTools", "Artifacts", "Base64", "CRC32c", "Dates", "Distributed", "Downloads", "FileWatching", "Future", "InteractiveUtils", "LazyArtifacts", "LibCURL", "LibGit2", "Libdl", "LinearAlgebra", "Logging", "Markdown", "Mmap", "NetworkOptions", "Pkg", "Printf", "Profile", "REPL", "Random", "SHA", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "StyledStrings", "SuiteSparse", "TOML", "Tar", "Test", "UUIDs", "Unicode", "CSV", "DataFrames", "Example", "JSON", "Crayons"])

The package evaluation job you requested has completed - no new issues were detected.
The full report is available.