Maybe you should consider to move the go source to a separate repository and reference it in this repository as a submodule or pull it while building. Separating a folder can be done with git filter-branch. Don't execute this command before reading the manual.

Also the go program schuld be in its own container.

Therefore it would be a good idea to separate the repository into a "go program repository" and references the "go program repository" here.