Obviously #320 suggests that we don't want to generate ES6 - that's fine. What I'm suggesting is to still generate standard JavaScript code, but with the assumption that SystemJS or browserify or webpack exists to fill in require. Even then this would be something that would happen only with a flag enabled.

A common package could be referenced for all of the common code that ends up at the beginning of the output file right now.

Right now DCE occurs based on the command being compiled. It may import io and only use certain things from there.

Another program may import io as well, but use more of its API.

So the two compiled io packages would not be equivalent nor interchangeable.

Do you think this could be a replacement for the existing code generation, or do you see it as an optional secondary mode that's enabled via a flag?

I don't have many thoughts on this so far (aside from comments above). I wonder what @neelance thinks.

My main concern/question is whether or not the benefits offered by doing this outweigh the added maintenance and support cost. I don't know enough about the benefits or the cost right now.

I have been thinking about something along these lines for a few weeks, but haven't said anything as the actual need hasn't arisen in my coding, so I haven't spent much time thinking through the implications.

My thought was that it could make for a convenient way to allow GopherJS "plugins". In my case, the end goal would be to allow simple Go methods, with no other boiler plate, to execute as JavaScript code. There are a number of applications where this is done with JavaScript (or probably any dynamic language) and expanding that to Go would be nice.

A couple examples that come to mind (although I'm sure there are countless others):

• CouchDB's map and reduce functions.
• Haraka plugins

One of my current projects would likely benefit from the ability to write "plugins" in Go, which, for my purpose, would require that the plugins themselves can be built as stand-alone *.js files to be later required on demand.

A common package could be referenced for all of the common code that ends up at the beginning of the output file right now.

Right now DCE occurs based on the command being compiled. It may import io and only use certain things from there.

I expect DCE would have to be turned off for any packages built this way.

If this feature is to be implemented, perhaps it makes sense to allow compilation of specific packages this way with a command-line flag which also disables DCE for that package. This way one could compile io without DCE, but other packages could be DCEd and bundled together with main.js.

Then we still need some way to tell GopherJS that the main.js build should exclude io. I'm not sure how that would look/work.

There are so many advantages to structuring this way, and I'm wishing for it every day now that I'm into rapid development with Gopher.

• Webpack can hot reload modules that changed. So, we could hot reload in development just the go packages that changed, with minimal effort from GopherJS (I'd assume)
• Webpack can just rebuild modules that changed. (tree shaking)
• Webpack can bundle things into different .js files depending on how your code is structured, and then lazy load them. If Gopher produced modular code, the browser could just load the Go packages it needs when it needs them. Go is already structured like this, it makes sense that we could generate Javascript that follows the pattern.
• I could make TypeScript typings that actually have correct scoping - import { IMyStruct } from '@gopher/github.com/myorg/mypackage';

Not to take away form what you're saying, just trying to understand better, but doesn't the gopherjs build tool already do most of those things? When you do gopherjs install or gopherjs serve, it already rebuilds only what has changed, it performs some DCE (which can be improved more so on the GopherJS side, may be harder outside), etc.

This seems like replacing one tool with another.

I do remember Richard discussing/considering webpack (IIRC) some time ago though. Still waiting on his feedback here.

@shurcooL It rebuilds only what changed from a Go standpoint, but from a browser bundle standpoint, you will need a full page refresh to get the changes.

I'm not using Gopher to bundle my entire app - that is to say, I'm using a lot of npm packages, angular2, some other things that Gopher can't be expected to bundle properly. That is the job of tools like Webpack, browserify, etc.

If the Gopherjs output was modular as we're describing here, that is to say, using relative require statements to pull in other packages when needed, tools like Webpack can understand the code structure better, and bundle things correctly when building the final app AND in development. In development right now, even if I change just one line of code in one Go package, Webpack sees the entire output file changed, and all of the Go code gets reloaded in the browser (without a refresh due to hot module reloading, but at the same time all of the Go code gets re-initialized). If Gopher was modular, Webpack could reload just the Go packages that changed, allowing for far faster Webpack builds, faster iteration in development (swap in just the package module that changed), and an overall better experience.

The point is that Gopher can't hope to be the compiler for an entire modern webapp, at least in most realistic situations where other js libraries with transpilers, compilation pipelines, static assets, loaders, etc are in play. It'd therefore be nice if Gopherjs produced output that the full-fledged tools to do this can understand.

Here are some problems that I see:

• This would break the Go spec, especially around package initialization. init functions and package variable initializations have to run in a specified order at application startup.
• There is some state in the GopherJS runtime which makes it hard, but not impossible, to hot-swap single packages.
• Any dead code elimination (tree shaking) that Webpack or some other JS tool does will always be much inferior to what GopherJS itself could do. This is because GopherJS has more detailed information available about the code than any JS tool can infer from reading the JS output.
• The output of a single package does not only depend on the package itself, but also on the exported types of its dependencies. This means that when you change some package, other packages that depend on it need to be recompiled, too.

Currently the output of GopherJS can be a single module for Webpack. If that module changes, then Webpack reloads that module. This should be very fast, since it makes no sense to further post-process the output of GopherJS, thus Webpack should nearly instantly reload that module in the browser. Initializers will get executed again, yes, how is that problematic in your case?

@neelance these things make sense and it did occur to me it might be extremely difficult to get hot swapping working.

The issue right now is that as the output of gopher is so massive it takes webpack something like 5-8 seconds to rebuild when it changes, and then the entire go codebase and everything that depends on it (right now that's an entire page of the site) has to reinit.

Everything still works ok doing this. It's not horribly inconvenient and it works, at least. In development it would be nice to have some kind of hot reload that didn't necessarily respect the Go spec for package init, but in exchange offers a far faster development cycle.

Might not be worth the effort. Not sure.

@paralin

... the output of gopher...

I think you'll find everyone refers to "it" as GopherJS, but that's a minor point :)

Everything still works ok doing this

I'd be interested to understand why the output from GopherJS has to go through Webpack at all? I'll ask that question by first admitting I know nothing about the "workflow" to which you refer. Surely the two can exist side-by-side?

As far as dead code elimination is concerned, I wonder, is the situation you're describing one where some non-GopherJS code is the entry point for the web application, where this code then makes calls into GopherJS code? That being the case, then I think we're looking at a different problem, specifically one of enumerating the various external uses of GopherJS code (that by definition GopherJS cannot know without some help)

For example, if you only required the ability to call math.Abs (chosen because math imports almost nothing) then, assuming GopherJS did some form of dead code elimination, all we require is some way of telling GopherJS that's all that can be called. An alternative way of telling GopherJS exactly the same thing is to declare a main function as such:

package main

import "math"

func main() {
    _ = math.Abs(5)
}

As @neelance pointed out above, I don't think it would make sense to delegate this dead code elimination task to webpack because GopherJS will always do a better job (assuming #186 or similar is implemented).

So to my mind this comes down to understanding:

• how to tell GopherJS about external references to Go declarations (albeit through their GopherJS-generated output)
• how GopherJS and webpack (or similar) can work alongside each other in terms of workflows etc, understanding that webpack does have a place here (I have a similar use case of using React, HighCharts etc...., but my entry point is GopherJS)

@myitcv It's completely understandable that GopherJS cannot hope to understand how it will be called externally. This is something that WebPack is really good at, it can do static analysis of your es6 module tree and figure out what depends on what, and then when necessary, rebuild what changed (by build I mean run the code through the plugin / import / load pipeline you define) and push only what changed out to the browser. I can understand if you don't want to delegate this task to an external tool, though.

They do work together, actually. I just import the generated GopherJS code as a module require('@myorg/goproject'), then in the Go main function, set module.exports to a wrapped object with constructor functions for the Go code I want to use from JS, and then call those functions from JS.

Then, when the Go code changes, webpack sees this change, and reloads everything that uses the Go code. So this means in the context of the web game I'm making, the Angular2 component that contains the <game-view> element that contains the instance of the Go code constructed with what I export from func main() is destroyed. This destroy call propagates all the way into Go, where my code cleans everything up, shuts down all running tickers, goroutines, etc... Then the code is swapped out in the browser, the element is re-inited, the Go code is called again (after a main() func is run due to the module swap), the constructor called again... and so on.

This works just fine, actually works very well - the only thing that would be better is if I could do iterative code development without reloading the entire Go program. Furthermore, if I could import Go packages like ES6 modules (think no main() func required to import something), WebPack could pull in only the Go packages in the import tree of the Go package I reference in my TypeScript/JS code.

The 2 main advantages are:

• Webpack can figure out what Go packages my code needs, relieving GopherJS the need to do this
• If each Go package was emitted as a separate file, we could hot swap in code when it changes. This is only something that happens in development, so breaking the Go spec and re-running all the init() stuff in that package when it is swapped in should be fine.

This kind of workflow would allow GopherJS to effectively mesh with the modern ES6 / modular code structure of browser apps. It's not exactly compliant with the Go spec in all ways - for example, you wouldn't have a main() function in each package (although this is one way I can imagine implementing this without changing GopherJS at all).

P.S: I know it would be extremely difficult right now to swap a package in given there may be sleeping goroutines in it, etc... I'm sure there's some kind of workaround for this, though, given enough thought.

@paralin

I can understand if you don't want to delegate this task to an external tool, though.

Just to clarify, I'm speaking on my own behalf as an interested party in GopherJS... @neelance and @shurcooL speak on behalf of the project.

The 2 main advantages are...

I think the point @neelance was making is that Go packages and a webpack modules aren't fungible; GopherJS encapsulates the logic required to enforce the Go spec and other Go-related matters that webpack doesn't (and will never?) know about because it operates at the module level. Hence, the output from GopherJS is a single module.

The one area that I think can be improved relates to how the dependencies of and on GopherJS code can be (automatically) declared, specifically:

1. the external dependencies of GopherJS code, i.e. the Javascript modules like React, called from within GopherJS code. This set of dependencies would be fed to webpack or similar
2. the GopherJS dependencies of non-GopherJS code. GopherJS would consume this set of dependencies (along with the package main entry point if there is one)

Set 1 would allow the single module output from GopherJS to be loaded as required according to Javascript module dependencies and also allow webpack to optimise loading etc as you better understand than me.

Set 2 would allow the aggressive dead code elimination described in #186 meaning that the single module output from GopherJS is as small as possible (again, to @neelance's point, GopherJS will always know better than webpack how to optimise this step)

@paralin 5-8 seconds for a rebuild are annoying, I agree with that. I'm wondering why it takes so long. Have you tried using https://webpack.github.io/docs/configuration.html#module-noparse and made sure that there are also no other processing steps being done on that file?

@neelance for the sake of follow up - yes I did, and that didn't seem to help.

The codebases that I was working on when writing this issue are now public, and could be a nice reproduction:

• http://github.com/paralin/Terram
• http://github.com/paralin/goterram
• http://github.com/paralin/gogame

Most important I think would be the feature of hot-reloading, so the game could keep playing while you edit something minor. (this is fine if this sometimes breaks running code, it's meant as a hacky development convenience). Also code-shaking, or eliminating dead code not used by callers of a module.exports style go library.

Hey all... Although not an exact fix for this issue, I'll post an intro here because it's connected:

GopherJS is an amazing tool, but I've always been frustrated by the size of the output JS. I've always thought a better solution would be to split the JS up by package and store it in a centralised CDN.

This architecture would then allow aggressive caching: If you import fmt, it'll be delivered as a separate file fmt.js, and there's a good chance some of your visitors will already have it in their browser cache. Additionally, incremental updates to your app will only change the package you're updating, so your visitors won't have to download the entire dependency tree again.

Today I'm announcing jsgo.io, which is this system. Here's how it works:

Visit https://compile.jsgo.io/<path> to compile or re-compile your package. Here's a very simple hello world - just click Compile: https://compile.jsgo.io/dave/jstest

After it's finished, you'll be shown a link to a page on jsgo.io: https://jsgo.io/dave/jstest. The compile page will also give you a link to a single JS file on pkg.jsgo.io - this is the bootstrap loader for your package. Add this URL to a <script> tag on your site and it will download all the dependencies and execute your package.

The compile server should be considered in beta right now... Please add issues on https://github.com/dave/jsgo if it's having trouble compiling your project. The package serving framework (everything in pkg.jsgo.io) should be considered relatively production-ready - it's just static JS files in a Google Storage bucket behind a Cloudflare CDN so there's very little that can go wrong.

Let me know what you think!

@dave nice work, what I think is that I'm going to store in IPFS instead of Google and deliver updates to the browser via p2p :)

The 5 – 8 seconds pauses during incremental development isn’t the only problem with putting all libraries in the same file. Billions of Internet users still have very slow Internet connections. In the other thread, I suggested that perhaps versioned renaming could be employed to solve the problem of matching DCE pared files to their counterpart files?

Webpack can bundle things into different .js files depending on how your code is structured, and then lazy load them. If Gopher produced modular code, the browser could just load the Go packages it needs when it needs them. Go is already structured like this, it makes sense that we could generate Javascript that follows the pattern

We are running into this problem. We'd like to use webpack code splitting to load the GopherJS-compiled module on demand and remove it from the main bundle, mainly due do the large output size. It seems webpack and the current Gopherjs output is not compatible and this issue might solve this.

It would be great to address this issue.

@benma it is possible to make GopherJS output behave like a single CommonJS module — it's mostly a matter of exporting the interfaces you want via js.Module. Though I haven't verified this myself, I think webpack should be able to split that out from the bundle.

Overall, I'd like GopherJS play more nicely with JavaScript ecosystem, but on my personal priority list this comes after a number of cleanups internal to GopherJS itself. But if anyone's interested in contributing this sooner, we'd very much welcome it :)

@nevkontakte thanks a lot! Using js.Module was necessary to make it work (we also had to upgrade from webpack 4 to webpack 5). Cheers!

I have having standard JS packages these days would work with all different tooling there is and require and other features do not even have to supported anymore.