WIP

I'm not sure if this idea (see below) will work yet. I'm building it up and will take this out of WIP when it is in a good state that we think will work. I'm going to switch to a different update and work on this in the background until I can finish the concatenation of AST files correctly.

Please comment anything that you think might help or you think would work better. Any feedback will help me refine this idea and the code.

Problem

While working on generics, one task is to share generics across packages so that correct instances can be created. This means that the AST files themselves need to be in memory at the same time and we have to delay converting them to Archives until after the instantiation of types. (This seemed way simpler than adding all the information to instantiate generics into the Archives and doing a second pass of analysis after filling in the missing instances.)

Since we need all the AST files at the same time, we can upgrade to use the Load method from x/tools/go/packages (packages) that will read all the packages in memory before returning. However, to perform our file augmentation and native overrides whilst using packages, we have come up with a way to perform the augmentation with the fields exposed in packages.Config.

Solution

This code change creates an on-demand augmentor that will fit into the ParseFile function pointer field of packages.Config. The idea is that when ParseFile goes to parse the file, we read and parse the file (probably with parser.ParseFile), then we augment it, before returning from ParseFile.

How we do that is to look at the package path of the file via the file's AST. If we haven't seen the package before we call it the "first file" for that package. If the package is one of the ones that we have overrides for, then the on-demand augmentor reads the overrides into additional file ASTs. We process additional ASTs to get the overrides. We apply those overrides to the first file and store them in the augmentor for later. We then append all the additional ASTs to the end of the first file. The Pos for the additional ASTs will still point to the virtual native override files in the FileSet so it will still report the correct location on a parse error. When another file from the same package is loaded, the augmentor will look up the overrides for that package and apply the overrides to that file.

This code change will not switch us over to use packages since that would require a lot of work in the builder and contexts. This is simply to get the augmentation to be performed on-demand like we will need it to be for packages, even though without using packages it seems like an odd move.