Thanks for sending this PR. I agree that the current cache busting is more conservative than it has to be, although in this case it's better to be slow than produce invalid results. The fix looks good overall, but I'll need a bit of time to think all implications through.

👍 Sounds great, thank you!

I definitely agree it's better to be slow and correct than fast and incorrect. I'm definitely not an expert on the go/gopherjs build system by any means -- I appreciate the consideration.

Unfortunately I found a case where this change would lead to incorrect results 😟 Consider package sort in the standard library. When we are testing it we build a test variant of sort package itself (with all *_test.go sources with package sort) and an "external test" package (all *_test.go sources with package sort_test). Turns out, the following dependency chain exists: sort_test → testing → regexp → sort [for sort.test]. Which means that regexp package built here has a potential of being different because it depends on a different variant of the sort package than if we do a simple gopherjs build regexp.

In practical terms this means that the compiledWithTests := testedPackage == importPath cache key condition is not sufficient. Instead, if should be compiledWithTests := importPath depends on testedPackage. Given the lazy approach gopherjs currently takes to loading imports, checking that condition is quite a bit more difficult.

I do have a long-term goal of making the build process consider build graph more explicitly (which would unlock smarter build caching strategies among other things), but there are quite a few steps to get there. #693 (comment) is the first of them, so I'll take this opportunity to advertise https://github.com/nevkontakte/gopherjs/tree/syscall2 branch and encourage early testing 🙃

🤯 that's a good catch. That answers my uncertainty here

In addition to this change, would it make sense to also add an IsForTest flag to BuildCache which is true when we have a TestedPackage? That would guarantee that gopherjs test and gopherjs build always use different caches. Then I think the compiledWithTests flag will always hold up.

(That said, I don't want to propose a change if you're just going to take things in a completely different direction and it's going to create more work for you that you'll just have to rip out. Perhaps my solution is just patching holes and opening us up to other issues rather than fixing the core problem)

I'll definitely check out your syscall2 branch tomorrow (we use darwin locally, so I'm excited to drop GOOS=linux from our test targets 😄 )

Unfortunately, a boolean IsForTest flag won't be sufficient either. Consider my example with sort_test above: when we are building regexp package for sort_test, it would be built with sort [for sort.test] as an input and cached with BuildCache{IsForTest: true}, compiledForTest = false. Then if we try testing net package, regexp will still be built (since testing depends on it) with the normal sort as an input and cached with the same BuildCache{IsForTest: true}, compiledForTest = false despite having different inputs compared to the first example.

And so we do really have to partition build cache by TestedPackage to ensure correctness, or get significantly smarter when analyzing build dependencies. The former is what we have now, the latter requires some significant refactoring to replace our use of go/build with golang.org/x/tools/go/packages. We'll get there eventually, but it will take some time. If you are interested in contributing towards that goal — let me know 😉

That said, I'm really surprised that the compilation itself increases your CI time by whole 10 minutes. In my experience test execution dominates build time by a good margin. Is your code base open source by any chance? It could be a good material to profile the compiler against and maybe find the root cause of the slowness.

👍 Gotchya, that makes sense 😞

I don't think we have the resources to contribute in a big way right now, but I can bring it up. I'm certainly interested in the health of gopherjs.

Unfortunately, our code base is not open source. We have a pretty large codebase (about 250k lines of production and test code each, not counting dependencies). If you think it would help, I can look into scrubbing "sensitive" output from the info logs during a run of gopherjs test and providing that.

FWIW, we do run the gopherjs tests in parallel with the rest of our build/tests, and this is not the bottleneck in our total CI time. I was just hoping this was a relatively easy fix, but clearly it's not 😆

Thanks for the offer, info logs gopherjs currently produces are not very helpful for performance analysis. The best option would be to capture a pprof CPU profile of the gopherjs tool. I thought I added a flag for that but apparently I didn't ¯_(ツ)_/¯ I guess we could park it until I have more time to look at performance and I might request your assistance then 🙂

As for this pull request, I think it is clear that it can't be merged without compromising output correctness, so I am going to close it. I do appreciate the effort and we will be happy to accept future contributions, big or small!

👍 Sounds great, I appreciate the guidance on this.