Yeah I should've provided a demonstration in the first place.

I've created https://github.com/kapple19/TempPkg which demonstrates this issue.

Package code containing error: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/src/TempPkg.jl#L1-L9

module TempPkg

public foo

function foo()
    return Bar(2) # Bar is intentionally undefined
end

end # module TempPkg

runtests.jl content: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/test/runtests.jl#L1-L4

using TempPkg
using ParallelTestRunner

runtests(TempPkg, ARGS)

JET call: https://github.com/kapple19/TempPkg/blob/d01da436f82ec16cfb1ad274729386c8df2be5e7/test/jet.jl#L1-L4

using TempPkg
using JET

JET.test_package(TempPkg; target_modules=(TempPkg,))

Output:

(TempPkg) pkg> test
      Status `~/.julia/dev/TempPkg/test/Project.toml`
  [c3a54625] JET v0.11.3
  [d3525ed8] ParallelTestRunner v2.5.0
  [7c2ae950] TempPkg v0.1.0 `.`
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/TempPkg/Manifest.toml`
  [da1fd8a2] CodeTracking v3.0.0
  [807dbc54] Compiler v0.1.1
  [b5f81e59] IOCapture v1.0.0
  [c3a54625] JET v0.11.3
  [aa1ae85d] JuliaInterpreter v0.10.10
  [70703baa] JuliaSyntax v1.0.2
  [6f1432cf] LoweredCodeUtils v3.5.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.0
  [bac558e1] OrderedCollections v1.8.1
  [d3525ed8] ParallelTestRunner v2.5.0
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [05181044] RelocatableFolders v1.0.1
  [295af30f] Revise v3.13.2
  [6c6a2e73] Scratch v1.3.0
  [10745b16] Statistics v1.11.1
  [7c2ae950] TempPkg v0.1.0 `.`
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
     Testing Running tests...
Running 1 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
jet        (1) │     1.55 │   failed at 2026-03-16T07:47:45.723

Test Summary: | Fail  Total  Time
  Overall     |    1      1  5.7s
    jet       |    1      1  1.5s
    FAILURE

Error in testset jet:
Test Failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))

ERROR: LoadError: Test run finished with errors
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/runtests.jl:4
ERROR: Package TempPkg errored during testing

But calling the testing code directly provides an error message:

(TempPkg) pkg> activate test
  Activating project at `~/.julia/dev/TempPkg/test`

julia> include("test/jet.jl")
Precompiling TempPkg finished.
  1 dependency successfully precompiled in 1 seconds
JET-test failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))
  ═════ 1 possible error found ═════
  ┌ foo() @ TempPkg /var/home/kapple/.julia/dev/TempPkg/src/TempPkg.jl:6
  │ `TempPkg.Bar` is not defined: TempPkg.Bar
  └────────────────────
  
ERROR: LoadError: Test.FallbackTestSetException("There was an error during testing")
Stacktrace:
 [1] record(::Test.FallbackTestSet, t::JET.JETTestFailure)
   @ JET ~/.julia/packages/JET/lX05m/src/JETBase.jl:1434
 [2] func_test(func::typeof(JET.report_package), testname::Symbol, args::Any; broken::Bool, skip::Bool, jetconfigs::@Kwargs{…})
   @ JET ~/.julia/packages/JET/lX05m/src/JETBase.jl:1407
 [3] top-level scope
   @ ~/.julia/dev/TempPkg/test/jet.jl:4
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/jet.jl:4
Some type information was truncated. Use `show(err)` to see complete types.

And just to confirm I'm not crazy:

(TempPkg/test) pkg> activate .
  Activating project at `~/.julia/dev/TempPkg`

(TempPkg) pkg> test
      Status `~/.julia/dev/TempPkg/test/Project.toml`
  [c3a54625] JET v0.11.3
  [d3525ed8] ParallelTestRunner v2.5.0
  [7c2ae950] TempPkg v0.1.0 `.`
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/TempPkg/Manifest.toml`
  [da1fd8a2] CodeTracking v3.0.0
  [807dbc54] Compiler v0.1.1
  [b5f81e59] IOCapture v1.0.0
  [c3a54625] JET v0.11.3
  [aa1ae85d] JuliaInterpreter v0.10.10
  [70703baa] JuliaSyntax v1.0.2
  [6f1432cf] LoweredCodeUtils v3.5.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.0
  [bac558e1] OrderedCollections v1.8.1
  [d3525ed8] ParallelTestRunner v2.5.0
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [05181044] RelocatableFolders v1.0.1
  [295af30f] Revise v3.13.2
  [6c6a2e73] Scratch v1.3.0
  [10745b16] Statistics v1.11.1
  [7c2ae950] TempPkg v0.1.0 `.`
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
     Testing Running tests...
Running 1 tests using 1 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
               │   Test   │ ──────────────── CPU ──────────────── │
Test  (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
jet        (1) │     1.55 │   failed at 2026-03-16T07:48:29.204

Test Summary: | Fail  Total  Time
  Overall     |    1      1  5.7s
    jet       |    1      1  1.5s
    FAILURE

Error in testset jet:
Test Failed at /var/home/kapple/.julia/packages/JET/lX05m/src/JETBase.jl:1375
  Expression: (JET.report_package)(TempPkg; toplevel_logger = nothing, target_modules = (TempPkg,))

ERROR: LoadError: Test run finished with errors
in expression starting at /var/home/kapple/.julia/dev/TempPkg/test/runtests.jl:4
ERROR: Package TempPkg errored during testing

It's the following message that is missing when running pkg> test,
but which is present when includeing the test file:

  ═════ 1 possible error found ═════
  ┌ foo() @ TempPkg /var/home/kapple/.julia/dev/TempPkg/src/TempPkg.jl:6
  │ `TempPkg.Bar` is not defined: TempPkg.Bar
  └────────────────────

Error messages from JET.jl not printed #116

Description

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Error messages from `JET.jl` not printed #116