feat(compute/build): support native go #995

Integralist · 2023-08-14T14:10:12Z

This PR adds support for native Go when using the latest SDK (0.2.0) and an updated Go starter kit (not yet released).
For screenshots and details see here

This PR is blocking: fastly/compute-starter-kit-go-default#8

This is NOT a breaking-release, even though we bump the constraints for tinygo and standard go (required for the new Go SDK). That's because we parse the go.mod to see if the user has an SDK version < 0.2.0 and if so we (at runtime) downgrade the constraint to >= 0.26.0 rather than use the new constraint of >= 0.28.1.

Original PR description/details below...

This PR switches from using TinyGo to native Go.

🚨 NOTE: This appears to be a 'breaking' change (and thus will require a major version release) because ultimately although the user can configure settings in their manifest to control the use of TinyGo, out of the box, the CLI will try and use native Go and that is going to break the experience for existing users. See update here

~~Users are able to continue using TinyGo by adding a [scripts.toolchain] into their fastly.toml manifest file:~~

[scripts]
toolchain = "tinygo"

Examples

⚠️ The only example that might require discussion is the last one.

1

The following example demonstrates a new user, with go1.21 installed, and no other non-standard configuration.

We can see that the native Go build command is used (as it now supports WASI).

We can also see that we display a message to the user informing them of what they need to do to continue using TinyGo (if they are trying to build a pre-existing project that previously would have used TinyGo).

2

The following example demonstrates the error the user will see if they have go1.21 installed but have [scripts.toolchain] set to TinyGo.

Notice that TinyGo itself doesn't yet support go1.21.

2 new user with go1 21 but tinygo toolchain

3

The following example demonstrates a new user, with go1.18 installed, who has [scripts.toolchain] set to TinyGo.

3 new user with go1 18 and tinygo toolchain

4

The following example demonstrates the error the user will see if they have go1.18 installed but have [scripts.toolchain] set to native Go.

Notice the CLI has an explicit constraint on toolchain = "go" using Go 1.21 and that the version of Go the user has installed doesn't yet support WASI.

5

The following example demonstrates a new user, with go1.21 installed, who has no [scripts.toolchain] set but does have a custom [scripts.build] configured to use native Go.

6

The following example demonstrates a new user, with go1.18 installed, who has no [scripts.toolchain] set but does have a custom [scripts.build] configured to use TinyGo.

Notice how although native Go is now the default behaviour, because the user has go1.18 installed, and TinyGo, and they've configured their custom build script to use TinyGo, the CLI does indicate a warning that the constraint wasn't met (and also that the user should add toolchain = "tinygo") but ultimately the build succeeded as they were using all the right tools and scripts.

The reason this is slightly problematic is because the toolchain constraint defaults to native Go (i.e. go1.21) and only changes to go1.18 when [scripts.toolchain] is configured. If that's not configured, like in this example, we don't then want to get into the business of trying to manually parse [scripts.build] to understand what a user has entered. We could try and do a glob search for "tinygo" but we don't know if a user has tinygo aliased or they downloaded it manually as tgo or whatever the situation might be. We can't presume to know the context of the custom build script.

Integralist · 2023-08-14T15:22:26Z

NOTE: I had an internal discussion with reviewer @joeshaw and after a lengthy back-and-forth about how to avoid a major release, we decided that a major release was probably the best approach (as none of the alternatives were clear winners and each approach had its own set of caveats).

Integralist · 2023-08-14T15:35:43Z

@joeshaw once you've merged fastly/compute-starter-kit-go-default#8 let me know so I can quickly retest this PRs changes to be sure compute serve and deployments work.

joeshaw · 2023-08-14T16:50:06Z

.fastly/config.toml

 [language.go]
-tinygo_constraint = ">= 0.26.0-0" # NOTE -0 indicates to the CLI's semver package that we accept pre-releases (TinyGo users commonly use pre-releases).
-toolchain_constraint = ">= 1.18"
+tinygo_constraint = ">= 0.26.0-0"       # NOTE -0 indicates to the CLI's semver package that we accept pre-releases (TinyGo users commonly use pre-releases).


I mentioned this in fastly/compute-starter-kit-go-default#8 but version 0.2.0 of the SDK brings the TinyGo requirement up to >= 0.28.1.

joeshaw · 2023-08-14T16:55:27Z

pkg/commands/compute/language_go.go

+// who don't have a `scripts.build` defined.
+//
+// NOTE: Only used if `toolchain = "go"` and if Go 1.21+ is installed.
+const GoDefaultBuildCommand = "env GOARCH=wasm GOOS=wasip1 go build -o bin/main.wasm ./"


I am not confident env is going to work on Windows, so it's worth checking this if CI doesn't already.

Yeah, I wasn't confident this would work on Windows but CI (which runs the full test suite) seems to think there were no issues 🤔

In general would we just remove the env and just prefix the build script with GOARCH/GOOS? Not sure if exec.Command needs env.

env is required because exec.Command doesn't run a shell. We could set the environment outside of the command-line, but that obviously complicates things if we want the end user control over things.

You can set the environment with https://pkg.go.dev/os/exec#Cmd.Environ , although that would require parsing the command line with https://pkg.go.dev/github.com/google/shlex or something to extract the environment variable settings.

@dgryski I think it's fine to set the environment without shlex because we know what the default values for those should be (i.e. they're already hard-coded into the binary GOARCH=wasm GOOS=wasip1). So I'll look at doing that on Wednesday when I'm back from PTO.

Once the user starts defining their own build script, then things are a little out of our hands but I imagine if someone wants to define their own build script, then they could just do an export directly in their shell as a workaround. I'm not sure in practice how many people are going to stray from the pre-defined default paths in the CLI (for either Go or TinyGo)

@Integralist Do you mind making make test pass -v to go test (or making a target that does)? It'd be good to see the verbose output of the tests in the CI build logs and give me more confidence that the Windows build is working as it should.

Running TEST_COMPUTE_BUILD=1 go test -v ./pkg/commands/compute on my Mac gives me:

=== RUN TestBuildGo/build_success build_test.go:333: Fastly API token not provided Fastly API endpoint: https://api.fastly.com | Verifying fastly.toml... ✓ Verifying fastly.toml | Identifying package name... ✓ Identifying package name | Identifying toolchain... ✓ Identifying toolchain INFO: Creating ./bin directory (for Wasm binary) INFO: No [scripts.build] found in fastly.toml. The following default build command for Go will be used: `env GOARCH=wasm GOOS=wasip1 go build -o bin/main.wasm ./` INFO: The standard Go compiler >= 1.21 now supports WASI. To keep using TinyGo set `[scripts.toolchain]` to 'tinygo'. INFO: The Fastly CLI requires a go version '>= 1.21'. Build script to execute: sh -c env GOARCH=wasm GOOS=wasip1 go build -o bin/main.wasm ./ INFO: Command output: -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- ✓ Running [scripts.build] | Creating package archive... / Creating package archive... ✓ Creating package archive SUCCESS: Built package (pkg/test.tar.gz)

Which also shows that the script is run inside a shell, so I think env is unnecessary, at least on Unix.

JakeChampion · 2023-08-14T19:44:52Z

Question: Is there some reasons about why toolchain is within scripts? I was surprised to see it within scripts as it is not like the other entries within scripts, which are meant to be commands ran within a shell such as the build and post_init fields.

Integralist · 2023-08-15T05:59:23Z

@JakeChampion I originally had it top-level but moved it to scripts as it felt out of place at the top level.

Also, as per https://developer.fastly.com/reference/compute/fastly-toml/ scripts is for...

Customisation options for the Fastly CLI build step.

So as this is customising the build process it made sense (to me) to have toolchain added to scripts.

triblondon · 2023-08-15T07:13:04Z

I'm quite confused about scripts.toolchain. I expect my build to be executed using scripts.build, and so I wonder at what point does the scripts.toolchain command get executed on the shell?

triblondon · 2023-08-15T07:14:10Z

Oh, catching up on your comment about the definition of "scripts" @Integralist... that's not how I interpret scripts, scripts are surely executable commands?

Integralist · 2023-08-15T13:04:12Z

@triblondon I'm happy to move this back to the top-level if you feel it's more appropriate there.

triblondon · 2023-08-15T14:03:49Z

I am not a Go developer, so feel free to override me if this is expected in the Go community. I would have a hard time understanding how toolchain interacts with build (and after reading the PR description I'm only 80% sure I get it).

Is it not possible to just change the default build command in the CLI? It seems to me that we are already going to switch tinygo users to go automatically, with a warning about adding toolchain if they want to opt out. So instead, couldn't we do this? If the user has no build script set, we say:

No build script found in fastly.toml.  Fastly now recommends the standard Go compiler to build 
Compute@Edge projects, because Go added support for WASI in version x.y.z.  If you were previously
using TinyGo, and want to continue to do so, cancel this build, add the following to your fastly.toml, 
then re-run this build:

[command]

If you are happy to use standard Go, press enter to proceed. Add a scripts.build value in your fastly.toml
file to suppress this message on future builds.

Integralist · 2023-08-16T13:27:00Z

The latest commit reflects the new direction we're going in as far as implementation is concerned.

1. Pre-existing project with go1.18

Pre-existing projects cloned from our Go original starter kit have no explicit built script specified.

This means we'll default to using TinyGo.

2 pre-existing project user still on go1 18

2. Pre-existing project with go1.21

Pre-existing projects cloned from our original Go starter kit have no explicit built script specified.

This means we'll default to using TinyGo.

But if the user decides to upgrade their Go version, then their project will fail because TinyGo doesn't (yet) support Go 1.21.

Thankfully, TinyGo's error message is very clear: it indicates to the user that their Go version is not supported and also what is supported.

1 pre-existing project user upgrades to go1 21

3. New project with go1.18

New projects, cloned from the new Go starter kit, will have an explicit build script specified.

This build fails because go1.18 doesn't support compiling to Wasm.

4. New project with go1.21

New projects, cloned from the new Go starter kit, will have an explicit build script specified.

This build suceeds because go1.21 DOES support compiling to Wasm.

5. New project with go1.18 and custom tinygo build script

The build succeeds because although we have the new Go starter kit bumped to use the new 0.2.0 SDK, the user's tinygo custom script will be using TinyGo not native Go (as per the logic/constraints defined in the latest commit of this PR).

6 new project with new sdk but go1 18 and tinygo build script

6. New project with go1.21 and custom tinygo build script

The build fails because we have used the new Go starter kit (bumped to use the new 0.2.0 SDK), and so the user's tinygo custom script will be using TinyGo not native Go and TinyGo < 0.29.0 (not released) doesn't support go1.21.

Thankfully, TinyGo's error message is very clear: it indicates to the user that their Go version is not supported and also what is supported.

5 new project with new sdk but go1 21 and tinygo build script

Integralist · 2023-08-16T13:38:39Z

@joeshaw @dgryski

        Build script to execute:
        	cmd.exe /C GOARCH=wasm GOOS=wasip1 go build -o bin/main.wasm ./
        
        
        INFO: Command output:
        --------------------------------------------------------------------------------
        'GOARCH' is not recognized as an internal or external command,
        operable program or batch file.
        --------------------------------------------------------------------------------

Looks like we'll have to go down the https://pkg.go.dev/os/exec#Cmd.Environ route.

This would mean not setting those environment variables in the build script command for Windows users.

Integralist · 2023-08-16T14:50:48Z

@joeshaw @dgryski last commit fixed Windows issue

joeshaw · 2023-08-16T16:48:46Z

@Integralist did env work previously?

joeshaw · 2023-08-16T18:05:54Z

It looks like env does indeed work on Windows, see https://github.com/fastly/compute-starter-kit-go-default/actions/runs/5882217274/job/15952358049 where I test it out.

I think we might want to not automatically set GOARCH/GOOS because there will be a variation in behavior for TinyGo. See tinygo-org/tinygo#3861.

pkg/commands/compute/language_go.go

pkg/manifest/file.go

joeshaw · 2023-08-16T19:53:20Z

because there will be a variation in behavior for TinyGo

Specifically, tinygo -target=wasi pretends to be GOARCH=arm GOOS=linux because the standard library didn't have real support for wasi, so some different code is selected due to those implicit build tags.

pkg/commands/compute/language_go.go

This reverts commit 242793a.

Integralist · 2023-08-18T09:50:03Z

@joeshaw env works fine for Linux but not Windows...

I'm not sure what problem exists in the test suite implementation that would cause it to fail for the tests but work in practice (as demonstrated by the CI job you linked to).

Integralist · 2023-08-18T10:15:26Z

@joeshaw looks like my tests neglected to include env 🤦🏻 🙂

So all is good now and ready for your final review.

Integralist · 2023-08-18T18:25:48Z

UPDATE: The use of env doesn't work on a real/fresh Windows laptop (likely GitHub Actions installing tools to make it work). So we're going to add [scripts.env_vars] to the Go Starter Kit and add support for that in this CLI PR. This will allow the CLI to set the environment variables required for standard Go to compile correctly, while also allowing a user to override the env vars with their own settings if they so choose.

I'm PTO for just over a week, so I'll wrap up this outstanding work when I return.

joeshaw · 2023-08-22T13:23:08Z

An additional feature request as part of this PR: we should ensure that the binary output is actually a wasm file. If you ran a fastly compute build on a project that supported scripts.build but not scripts.env_vars you would get a native binary written to bin/main.wasm, because GOARCH and GOOS aren't set.

There's nothing we can do about this for existing versions (and fastly compute serve and fastly compute publish fail with appropriate error messages), but going forward this would be a helpful validation step.

The binary format (including the magic and version info at the start of a file) are defined here: https://webassembly.github.io/spec/core/binary/modules.html#binary-module

Integralist · 2023-08-29T17:15:11Z

@joeshaw OK I've implemented the scripts.env_vars and the Wasm validation logic.

Have another review and give this PR an official 'approval' if you're happy with it.

Thanks!

pkg/commands/compute/language_go.go

pkg/commands/compute/language_toolchain.go

joeshaw · 2023-08-29T18:13:06Z

pkg/commands/compute/language_toolchain.go

+		magic = magic[1:] // Remove the first byte which is a null character "\x00"
+	}
+	if string(magic) != "asm" {
+		return bt.handleError(err)


err will be nil in this case, so you'll need to construct your own error here.

This still needs addressing. err will be nil here.

EURGH! Sorry 😞

Just pushed a change.

pkg/commands/compute/language_toolchain.go

Integralist · 2023-08-30T10:55:27Z

@joeshaw one more review if you can please 🙂 🙇🏻

dgryski · 2023-08-30T18:39:01Z

pkg/commands/compute/language_go.go

+// NOTE: The `configConstraint` is the latest CLI application config version.
+// If there are any errors trying to parse the go.mod we'll default to the
+// config constraint.
+func identifyTinyGoConstraint(configConstraint, fallback string) string {


If this turns out to be too fragile, we can switch to https://pkg.go.dev/golang.org/x/mod/modfile

Cool thanks. I think our use case is simple enough but yeah if in the real-world it's starts being a problem we can pick up a tool like that 👍🏻

joeshaw

thank you for your persistence on this. 👍 LGTM

Integralist · 2023-08-31T08:22:18Z

thank you for your persistence on this. 👍 LGTM

@joeshaw thank you for all the great feedback and patience with my silly mistakes 🙂

I'm just sorry this took so much of your time to review, so I appreciate your efforts here.

Integralist requested review from dgryski and joeshaw August 14, 2023 14:10

Integralist added the enhancement New feature or request label Aug 14, 2023

joeshaw reviewed Aug 14, 2023

View reviewed changes

Integralist force-pushed the integralist/native-go branch 2 times, most recently from c08f58c to 1fda128 Compare August 16, 2023 13:18

Integralist force-pushed the integralist/native-go branch 2 times, most recently from 242f8d4 to 78cc351 Compare August 16, 2023 14:33

Integralist mentioned this pull request Aug 16, 2023

bump SDK to version 1.0.0 fastly/compute-starter-kit-go-default#8

Merged

Integralist requested a review from joeshaw August 16, 2023 16:18

joeshaw reviewed Aug 16, 2023

View reviewed changes

pkg/commands/compute/language_go.go Outdated Show resolved Hide resolved

pkg/commands/compute/language_go.go Outdated Show resolved Hide resolved

pkg/manifest/file.go Outdated Show resolved Hide resolved

dgryski reviewed Aug 16, 2023

View reviewed changes

pkg/commands/compute/language_go.go Outdated Show resolved Hide resolved

Integralist added 4 commits August 18, 2023 09:34

feat(compute/build): support native go

d9e466f

refactor(compute/build): clarify toolchain configuration

4b1062e

fix(compute/build/go): remove env from default build script

ab5430e

feat(scripts): add tinygo default starter kit

0e10b66

refactor(compute/build): replace native with standard

5efa4a6

Integralist force-pushed the integralist/native-go branch from bfb9747 to 5efa4a6 Compare August 18, 2023 09:18

Revert "fix(compute/build): support env vars for Windows"

0dadb5a

This reverts commit 242793a.

fix: add env to test build script

83340ff

Integralist requested a review from joeshaw August 18, 2023 10:15

Integralist added 4 commits August 29, 2023 15:01

feat(manifest): support scripts.env_vars

ebb7a16

doc(RELEASE): add comment about config_version

d5d1c0e

doc(RELEASE): remove internal notes for releasing

96f844b

feat(compute/build): validate wasm binary file

268362e

joeshaw reviewed Aug 29, 2023

View reviewed changes

pkg/commands/compute/language_toolchain.go Outdated Show resolved Hide resolved

joeshaw reviewed Aug 29, 2023

View reviewed changes

Integralist added 5 commits August 29, 2023 19:31

refactor(config): replace inline value with config value

5609539

refactor(compute/toolchain): move value to constant to avoids gosec G304

9f8d185

refactor(compute/toolchain): replace string parsing with bytes.Equal

80b11a9

refactor(compute/toolchain): use Read method from binary package

befafbe

feat(config): support resetting config to static embedded version

beec623

Integralist requested a review from joeshaw August 30, 2023 10:55

dgryski reviewed Aug 30, 2023

View reviewed changes

fix(compute/toolchain): create custom error

63704ce

joeshaw approved these changes Aug 30, 2023

View reviewed changes

Integralist merged commit be9cf61 into main Aug 31, 2023

Integralist deleted the integralist/native-go branch August 31, 2023 08:22

feat(compute/build): support native go #995

feat(compute/build): support native go #995

Uh oh!

Conversation

Integralist commented Aug 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Examples

1

2

3

4

5

6

Uh oh!

Integralist commented Aug 14, 2023

Uh oh!

Integralist commented Aug 14, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JakeChampion commented Aug 14, 2023

Uh oh!

Integralist commented Aug 15, 2023

Uh oh!

triblondon commented Aug 15, 2023

Uh oh!

triblondon commented Aug 15, 2023

Uh oh!

Integralist commented Aug 15, 2023

Uh oh!

triblondon commented Aug 15, 2023

Uh oh!

Integralist commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Pre-existing project with go1.18

2. Pre-existing project with go1.21

3. New project with go1.18

4. New project with go1.21

5. New project with go1.18 and custom tinygo build script

6. New project with go1.21 and custom tinygo build script

Uh oh!

Integralist commented Aug 16, 2023

Uh oh!

Integralist commented Aug 16, 2023

Uh oh!

joeshaw commented Aug 16, 2023

Uh oh!

joeshaw commented Aug 16, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joeshaw commented Aug 16, 2023

Uh oh!

Uh oh!

Integralist commented Aug 18, 2023

Uh oh!

Integralist commented Aug 18, 2023

Uh oh!

Integralist commented Aug 18, 2023

Uh oh!

joeshaw commented Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Integralist commented Aug 29, 2023

Uh oh!

Uh oh!

Uh oh!

Integralist commented Aug 14, 2023 •

edited

Loading

Integralist commented Aug 16, 2023 •

edited

Loading

joeshaw commented Aug 22, 2023 •

edited

Loading