I thought I would try my hand at this again today, and came to the conclusion that it's not really a straight-forward problem to solve. I think some discussion will be necessary to decide how it should even be approached.

Imagine an app, foo.go, which depends on a.go and b.go. a.go requires* JS module a, and b.go requires* JS module b. (This requiring might happen via js.Global.Call("require", "a"), or might be included in *.inc.js files--either should have the same result for this situation.)

The question is, where should node look for the a and b modules?

When I first filed this bug, my assumption was that it should look in the node_modules/ directory of the GopherJS module's root. But this would mean multiple directories in this scenario, and there's likely a good chance that some of them won't have node_modules populated anyway (at least there's no GopherJS infrastructure to insure that it is).

Assuming a.go and b.go are upstream modules, they would typically be installed via:

go get -u -d -tags=js github.com/foo/a
go get -u -d -tags=js github.com/bar/b

This would not (and probably should not), in any way, populate the respective node_modules directories.

So where does that leave us?

There are (at least) 3 scenarios to be dealt with:

1. gopherjs build

   In this case, arranging for inclusion of the files seems unimportant. We have no guarantee that node will even be the runtime environment, and it's well beyond the scope of GopherJS to configure Browserify shims, etc. All that is required is the output .js file. It is up to whomever is building the package to, at that point, configure the runtime environment.

2. gopherjs run

   Now GopherJS is controlling/configuring the runtime environment, and it seems like a reasonable expectation that calls to require ought to work. But how? There is the potential for depending on multiple GopherJS libraries which in turn have different node module requirements. Can we honor a user-provided package.json to manage this? Is it appropriate to add hooks to gopherjs to call npm install and related commands when gopherjs run is executed? Or is it reasonable to expect/require the users to handle fetching and installation of node packages on their own?

3. gopherjs test

   This is a special case or subset of gopherjs run. We still have the potential for dependencies on multiple GopherJS modules which in turn depend on multiple node modules. It might be that handling run would solve all the hard parts of test, too.

As a GopherJS coder, I suppose what at present makes sense to me, would be the ability to provide a package.json (or similar), which defines the npm dependencies of a given GopherJS module. Then the parent build process (gopherjs build or gopherjs test) would build a composite tree of all of these dependencies (probably handing that task off to npm) and install the necessary dependencies at build time, so that they are available to the nodejs process.

But I may be suffering from tunnel vision, and may be missing something obviously better. Are there other thoughts on how to accomplish this, or problems with my understanding?

#333 is a partial fix for this issue, and I believe it is adequately tested to be merged as soon as someone else signs off on it.

Still outstanding, and to be addressed in future PRs:

1. The security concern outlined above. The only fool-proof solution I've thought of is not to execute node code from /tmp at all (since node by default searches every parent directory until a match is found). Maybe we don't care to be this paranoid, though?
2. Handle dependencies of dependencies. I'm thinking of perhaps adding an npm hook to gopherjs, perhaps to be run as an optional command/switch (I'm not exactly sure what that would look like... I'd want something consistent with Go standard tools/extensions). This switch/option would:
   1. Search the source directories of any Go package to be included, looking for package.json files
   2. Extract the dependencies and devDependencies sections of each packages.json file to build a combined, temporary packages.json file
   3. It would execute npm install, feeding it the temporary packages.json file, to install all dependencies into node_modules, where node expects to find them.

I think a page should never load more than one file generated by GopherJS. For one, you would duplicate all dependencies (especially core packages) that could be shared. Also those two modules could only talk to each other via JS API, not Go API.

Instead, GopherJS libraries should simply be installed via gopherjs get and then the main package can include whatever it needs via plain Go import declarations. We may even want to actively discourage people from publishing GopherJS libraries via npm because of those reasons (@shurcooL What do you think?).

@flimzy I think you are right that gopherjs run and gopherjs test should use temporary files in the current working directory. That way node can behave as expected.

I think a page should never load more than one file generated by GopherJS. For one, you would duplicate all dependencies (especially core packages) that could be shared. Also those two modules could only talk to each other via JS API, not Go API.

I agree.

We may even want to actively discourage people from publishing GopherJS libraries via npm because of those reasons (@shurcooL What do you think?).

I think it is reasonable to discourage this use case, but it may not be possible (and probably neither necessary) to outright prohibit it.

Some day someone may write JS bindings for a Go library. That's not a situation that GopherJS currently makes "easy" (by virtue of the fact that a 'main' package is necessary to build a GopherJS package), but it certainly is possible.

If then someone else were to write GopherJS bindings for that node library, we'd end up double-including all of those dependencies... hopefully that situation will never arise, though.

@flimzy I think you are right that gopherjs run and gopherjs test should use temporary files in the current working directory. That way node can behave as expected.

If this is an acceptable solution, it probably would be cleaner, and would simultaneously address the security concerns. My first half-attempt at that method had some problems, but they should be easy to overcome. I'll work on that PR.

There's still the outstanding issue of multiple Go dependencies, each with node dependencies. (I'm not sure if your comment was meant to address that or not... let me spell out that scenario just in case it's not clear).

src/
    github.com/userX/
        foo/
            foo.go
            package.json
            foo.inc.js      // require('node-module-foo');
    github.com/userY/
        bar/
            bar.go
            package.json
            bar.inc.js      // require('node-module-bar');
    github.com/userZ/
        baz/
            baz.go          // import(
                            //      "github.com/userX/foo"
                            //      "github.com/userY/bar"
                            // )

In this scenario, we need to npm install node modules from two independent Go packages when building baz. Hopefully my 3-step approach above makes sense in this context.

I think it's not the job of GopherJS to care that much about npm. It should just compile the Go source, while allowing a setup that works with npm ("Do less, enable more."). In the case above, I would expect the main application github.com/userZ/baz to maintain a package.json that includes all necessary node modules, including those of the other Go packages that it uses. This is simple to understand and not that much of a burden (because hopefully Go code will not have that many dependencies as JS code usually has). If it really becomes annoying to maintain this list, then some external tool could provide the 3-step functionality you described, but not GopherJS itself.

That seems quite reasonable... and I was starting to come to a similar conclusion myself after thinking about what I had written yesterday.

I agree with the "do less and enable more" part. :P

I also think GopherJS itself should be just a compiler. The gopherjs should do as little as needed, and hopefully one day the GopherJS compiler can be used via standard go binary.

Fixed with 158ce0b

Is this really fixed? The tests still use the temporary directory.

No it's not. I was confusing this issue with #306, which was already closed.

But as I was considering the implications of this, I think it's more than a minor inconvenience. It could be a potential security concern, as a malicious user of the same system might put a phony 'pouchdb' library in /tmp, and the next time I run my test...

I'm trying to better understand this issue, so I have questions/comments. As far as I can tell, the above is not quite true. When ioutil.TempFile is called, it doesn't just use /tmp directory. From its docs:

Multiple programs calling TempFile simultaneously will not choose the same file.

If you look at its implementation, it uses the current time in nanoseconds and process ID to seed a random number generator when coming up with a random filename.

Ok, I think I see the problem. It's because ioutil.TempFile is used rather than ioutil.TempDir, and you're potentially installing npm dependencies in /tmp which is shared across multiple go test runs and is vulnerable to the attack you've described.

Is it possible to either use ioutil.TempDir to create a directory specific to the given test run, to avoid reusing /tmp for npm dependencies?

I think the ideal solution to this problem would be to continue to use a temporary dir in /tmp for files, but keep the current working directory as the current working directory of the test process/npm. Is that what you tried to do in #333? What was the problem there?

Sorry if I'm suggesting things that won't work, but I'd really like to understand the problem and dismiss potential solutions for concrete reasons before settling to using cwd for writing temporary files. IMO that's best to avoid if possible.

Final question, what are the implications of writing temp files to cwd? I see they're removed afterwards, right, so it would just be during a go test run, is that right? What are other known disadvantages of not using system temp folder?

I don't think the difference between ioutil.TempFile and ioutil.TempDir is meaningful in this case.

The core issue is that nodejs doesn't look only in the current directory--it also considers all parent directories (see here). So if the working directory or any parent directory is writable by some untrusted user, there is a potential for abuse.

So, if we use /tmp or any subdirectory of /tmp to execute our generated GopherJS code, nodejs's module search path will include /tmp/node_modules, which could be created by an untrusted user.

In a "normal" nodejs environment this is not, I suppose, considered a security issue, because nodejs packages are typically installed in secure locations, where parent directories are typically only writable by the owner of the file, or root (or perhaps some trusted group).

My patch builds and executes the packages in the same directory where the Go package lives. This works fine, as long as we have write access on this directory, but we don't in the case of the upstream Go tests. And we might not in other scenarios as well (I'm not sure).

Perhaps an alternative would be to use some sort of per-user temp directory. (~/.gotmp? $GOROOT/tmp?), which will not have a parent directory writeable by untrusted users. I wouldn't count on this working everywhere, so we'd probably still need to support an environment variable for users who can't rely on such a default for whatever reason.

Re: #333, yes, that is an attempt to leave the temp files where they are in /tmp/*, but modify the node environment to search the Go package's directory. But it's only a partial solution, because it doesn't prevent node from looking in /tmp, too. I'm not sure how easy it will be to get node to not look in /tmp. But if that's the path we want to take, I'm happy to investigate.

Final question, what are the implications of writing temp files to cwd? I see they're removed afterwards, right, so it would just be during a go test run, is that right? What are other known disadvantages of not using system temp folder?

As mentioned above, it breaks when we don't have permission to write to cwd. The only case I've seen this happen is with the upstream Go tests. But perhaps it could happen in other situations (??).

It may seem "sloppy", I suppose, to create temp files in cwd, but given that we already expect any build process to write output files there, I wonder if anyone would consider it a problem to write "temporary" output files there? Perhaps this is a matter of opinion more than anything??

(I hope I've addressed each of your questions now)

You have addressed most/all of my questions, thanks for providing that information.

The core issue is that nodejs doesn't look only in the current directory--it also considers all parent directories (see here). So if the working directory or any parent directory is writable by some untrusted user, there is a potential for abuse.

So, if we use /tmp or any subdirectory of /tmp to execute our generated GopherJS code, nodejs's module search path will include /tmp/node_modules, which could be created by an untrusted user.

In a "normal" nodejs environment this is not, I suppose, considered a security issue, because nodejs packages are typically installed in secure locations, where parent directories are typically only writable by the owner of the file, or root (or perhaps some trusted group).

I see.

So it's not a problem when using cwd, even though one of the parent folders might be /home or /Users, because it won't actually look inside other user's folders, only in /home/node_modules or /Users/node_modules.

But it is a problem when using any subdirectory in /tmp.

Is the reason for that being a potential risk because you might not have all required dependencies available, so it would end up looking in parent folders (and be satisfied by an unexpected alternative package; instead of failing to compile altogether because the dependency is missing)?

It may seem "sloppy", I suppose, to create temp files in cwd, but given that we already expect any build process to write output files there, I wonder if anyone would consider it a problem to write "temporary" output files there? Perhaps this is a matter of opinion more than anything??

That's a good point, go build does write to cwd. Although it's slightly different because it happens when user runs the command, and the user has the intention of writing that file to cwd. When running tests, you just want to see the test output, you're not intending to write to cwd.

I think if there's no way to reliably resolve security and other concerns with /tmp, then cwd is an acceptable fallback plan.

Re: #333, yes, that is an attempt to leave the temp files where they are in /tmp/*, but modify the node environment to search the Go package's directory. But it's only a partial solution, because it doesn't prevent node from looking in /tmp, too. I'm not sure how easy it will be to get node to not look in /tmp. But if that's the path we want to take, I'm happy to investigate.

I think it's worth investigating, but like I said above, if there's no way, then cwd is fine. It'd be a shame to use cwd for temp files if there's some way not to, IMO.

But it is a problem when using any subdirectory in /tmp.

Is the reason for that being a potential risk because you might not have all required dependencies available, so it would end up looking in parent folders (and be satisfied by an unexpected alternative package; instead of failing to compile altogether because the dependency is missing)?

That's exactly right. Any missing dependency has the potential to be provided by an untrusted user.

And lest we be tempted to think it's a small problem, let me suggest that I think it's virtually guaranteed that many packages will be built with missing node dependencies. Simply executing 'gopherjs build' before 'npm install' is all it would take. And unless GopherJS is going to start trying to manage/enforce node/npm requirements (which I think we all agree is beyond the scope of GopherJS), it's bound to happen--probably frequently.

But if that's the path we want to take, I'm happy to investigate.

I think it's worth investigating, but like I said above, if there's no way, then cwd is fine. It'd be a shame to use cwd for temp files if there's some way not to, IMO.

I'll try to look into this some time this week, and see how possible/messy it would be to approach things this way.

I dug into this a bit over my lunch hour. There's not an easy way to modify node's environment such that it won't look in /tmp when executing from there. The internal list of paths is built based on argv[1](i.e. the full path of the .js script being executed).

This internal list of paths is accessible, from within the nodejs process, as part of the process data structure, specifically process.mainModule, which looks something like this:

Module {
  id: '.',
  exports: {},
  parent: null,
  filename: '/tmp/x.js',
  loaded: false,
  children: [],
  paths: [ '/tmp/node_modules', '/node_modules' ] }

It may be possible to modify this data structure from within the JS, but before require() is called. We could replace paths with our own generated list of paths relative to the Go project, rather than relative to /tmp/*. However, putting that sort of logic into the GopherJS output seems pretty messy to me, not to mention that this level of meddling in a process's internal data structure seems at least semi-evil in general. There would also the problem of detecting when to trigger this logic (only when run from gopherjs test I imagine), and when to ignore it.

So after all this, writing temp files to the CWD looks like a simpler solution to me. What do you think?

I agree.

So where does that leave us? Do you have any bright ideas to avoid the environment variable in my PR?

No, I think we should proceed as is. I can't think of any better way right now, and all the constraints seem to suggest this is the only way to resolve the underlying issue.

Perhaps after using the new approach for some time, we'll have ideas for improvements, or perhaps it'll turn out not to be something on anyone's minds. In either case, possible future followup changes can be seen as optional enhancements to this solution.