Support Go Modules builds. #1042
Conversation
|
Didn't look at the code, but this might be useful: tinygo-org/tinygo#941 |
|
Thanks! I actually did look at that exact PR at one point, and materializing GOROOT was behind the two of my first attempts to support Modules (this PR representing the third). This did, however, prove somewhat tricky to get right:
You are welcome to check out https://github.com/nevkontakte/gopherjs/tree/go-tool/cmd/gopherjs-ng which has some of those experiments if you are curious. I still like the idea of materializing augmented GOROOT on disk, and I'm going to continue working on it, but I felt like we need a working support for Modules before 1.17, which was rumored to deprecate GOPATH entirely. This is PR is the simplest solution to the problem I found, but it also doesn't address some of the issues which make our build system so complicated and weird. |
|
Also worth mentioning that initially |
|
This seems to fail for me when trying to build a project containing the |
|
I've fixed the panic, which was triggered when expanding patterns containing "/...". I also replaced the gotool package with an actual go tool invocation, since the package predated go modules and didn't know how to match them anyway. |
Correct for TinyGo. I looked at what GopherJS did and intentionally did not do that. I made sure that only whole packages would be replaced, to simplify the build system. It makes working on the standard library for TinyGo a bit harder, but not by much. And it makes the build system a lot simpler.
That proposal seems like a very sensible thing to do! What I do in TinyGo is that for every supported target that isn't in Go (that is, WASI and the various baremetal systems) I pick a single GOOS/GOARCH pair for the standard library to use. So for example, for WebAssembly in the browser (which is supported by the Go standard library) I use GOOS=js GOARCH=wasm as usual, but for WASI (which isn't supported by the Go standard library) I use GOOS=linux GOARCH=arm. This works very well in practice, only the syscall package and some packages that interact with the runtime (net, os, reflect, ...) need to be replaced for that to work. This means that most code in existence will think it's a different OS/arch, but in practice it'll usually run just fine. In the (few!) edge cases where it doesn't, they can use the
We currently call |
|
I think GoScript is getting very much off topic. I'm happy to discuss this in a different place. |
|
@aykevl thanks for your detailed comment, all of what you said makes sense. In general I have high hopes that a lot of GopherJS standard library overlays could be simplified if we try to reuse wasm implementations from the upstream. That said, there's a significant challenge that is unique to GopherJS: we don't have access to the raw memory. So we have to patch any library that does pointer arithmetics, which means it's considerably more packages than @paralin I would agree with @aykevl that this discussion would be straying quite far from this PR's subject, so how about we continue it in #gopherjs channel in the Gopher Slack? I like the idea of goscript, but only after having worked on GopherJS compiler for a few months I began to appreciate how many subtle challenges there are if you start considering features like goto, goroutines and runtime packages. |
|
@nevkontakte Getting the following when importing syscall/js - that package is supported for compat w/ wasm code, right?
Thanks for the work on this |
|
@paralin I can't reproduce the error with this minimal example: https://gist.github.com/nevkontakte/56c18ec181a10881ddd84975bd8bc782 Commands I run: |
|
@nevkontakte repro here: https://github.com/paralin/gopherjs-repro-1042 just run "go run ./" in the root dir, or run "gopherjs build ./" in the app dir. |
…support. This commit introduces an abstraction for the go/build.Context called XContext, which will encapsulate GopherJS's customizations to the package loading and build process. In future, things like stdlib augmentation, embedded core packages, etc. will be hidden behind the XContext type. Chaining separate contexts for accessing packages on the real FS and embedded VFS is one of the prerequisites for using modules support go/build package has. Since it invokes `go list` command to obtain module-related information, we can't override FS access methods the way we were doing previously. So instead we chain two contexts for accessing packages on FS and VFS respectively.
For packages that come from a module, go/build returns PkgObj path inside the module root. Using it as-is leads to mixing source code and binary artifacts and version control mishaps. We solve the problem by detecting such paths and storing them in a separate build cache directory. Note that our implementation is much simpler and not compatible with the Go build cache. We rely upon go/build returning unique PkgObj paths for different packages and simply hash the original path to produce a path inside the cache.
go/build disables its module support (arguably incorrectly) whenever ReleaseTags are set to anything other than default. While such situation should be rare in practice, GopherJS formally supports being built by a version other than it's compatible Go version, and this hack ensures module support doesn't mysteriously disappear in such cases.
Under Go modules pkg.Dir may be initialized as a relative path whenever a build target is references by a relative path. For consistency, we convert them all to absolute paths. This also resolves an issue where "gopherjs build ." creates output named "..js".
gotool package matching logic was based on the old version of go (most likely 1.8 or so), and its patterns were not matching packages from third-party modules. We use `go list` instead to match packages on the physical file system. We also provide a shim for VFS-based build contexts using buildutil package, which should be good enough for the few cases we have.
Under Go Modules package sources a no longer under a single GOPATH and import paths can't be mapped onto the file system naively. Instead we have to import the correct package and open the file relative to package root. Since we don't know which part of the requested path represents the package import path, and which is a file within the package, we have to make a series of guesses until one succeeds.
Go modules will now be used for both gopherjs build and test execution by default. GOPATH mode is considered deprecated and there's little value in spending limited CI resources to test it in parallel with Modules mode.
Vendored directory is not supported outside of module root under Go Modules. I don't think we really need to test this specifically for GopherJS since vendoring support is completely provided by go/build and we indirectly use it when we build standard library anyway.
Vendoring test layout requires GOPATH build mode, though we no longer use it in the CI workflow by default.
We use https://github.com/gopherjs/todomvc as a simple test case for either mode, since it is small and easy to debug, but still has a few external dependencies to exercise Modules infrastructure. I also added a step that verifies that the output in both modes is identical. Strictly speaking this is not a requirement, but the current implementation maintains this invariant, so we might as well test it.
We do want to load it fully from the VFS, since we completely reimplement the package.
go list only returns this package with GOOS=js GOARCH=wasm. I've verified that no packages are added or removed from the test set by this change.
7d3da09 to
39c1917
Compare
|
@nevkontakte thanks for looking into it & doing the work to implement modules with gopherjs! |
GopherJS gained support for building code in module mode in 2021 (see PR #1042), and it itself didn't have problems being built in module mode since much earlier. Recently, it started getting semver tags that are recognized in module mode like `v1.17.2` and `v1.18.0-beta1` (see issue #847). Start suggesting the `go install` command to install its latest release in the README. (It remains possible to get the latest development version with something like `go install github.com/gopherjs/gopherjs@HEAD`, or one of the pre-release versions from `go list -m -versions` such as `go install github.com/gopherjs/[email protected]`.) While here, also update to the new shorter Go website domain and latest Go 1.18 point release.
Fixes #855.
I took the approach of leveraging module support that's built into the
go/buildpackage. Doing so has two prerequisites:go/build.Contextinstance must not have its file system methods overridden.Most of the changes in this PR tend to the #1. We were overriding FS access methods in order to make packages embedded in the VFS available during the build. Now we use two separate contexts chained together to access packages on the real FS and the embedded VFS. This refactoring is useful in the long run to make package loading logic more encapsulated in the build package.
The second prerequisite is somewhat bogus for reasons explained in the
build/versionhackpackage. golang/go#46856 is filed in the upstream for a better behavior, but until it's fixed, we will just work around it.