Thanks to visit codestin.com
Credit goes to github.com

Skip to content

compiler: use hash calculation for determining archive staleness #805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
myitcv wants to merge 12 commits into
base: master
Choose a base branch
from
Open

compiler: use hash calculation for determining archive staleness #805

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e807fdb
compiler: use hash calculation for determining staleness
myitcv Apr 23, 2018
e615455
Add --bv flag to test to log verbose build context messages
myitcv Apr 23, 2018
69257f7
Add build tags to the hash
myitcv Apr 23, 2018
f6baa10
More readable version of hash for debug purposes
myitcv Apr 23, 2018
41c38df
Fix file closing
myitcv Apr 23, 2018
d73d4bb
Address feedback from hajimehoshi
myitcv Apr 23, 2018
9c06cba
Refactor to prefer defers for Close
myitcv Apr 23, 2018
82f8b60
Further refactor to use defer for close
myitcv Apr 23, 2018
2e40249
Use better key name for gopherjs binary hash
myitcv Apr 23, 2018
3d5665a
Move compiler hash calc to init phase
myitcv Apr 24, 2018
5c358e3
Remove --bv flag on test
myitcv Apr 24, 2018
1ed4e6a
Add basic test of new hash staleness
myitcv Apr 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move compiler hash calc to init phase
  • Loading branch information
@myitcv
myitcv committed May 4, 2018
commit 3d5665a12bb1298f0fa5e9b85caf7cec64d78926
55 changes: 30 additions & 25 deletions build/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,25 @@ import (
"golang.org/x/tools/go/buildutil"
)

const (
hashDebug = false
)

var (
compilerBinaryHash string
)

func init() {
// We do this here because it will only fail in truly bad situations, i.e.
// machine running out of resources. We also panic if there is a problem
// because it's unlikely anything else will be useful/work
Copy link
Member

@flimzy flimzy Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than panicking here, would it be useful to fallback to a cache-disabled state?

Maybe that would allow running the gopherjs compiler in unusual environments (such as within a browser?)? (That may already be impossible for other reasons, idk).

Copy link
Member Author

@myitcv myitcv Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a fair point. I think I'd like to understand that situation a bit better; because a whole load of logic in build depends on the OS being available (writing archives etc).

The compiler is run within the browser by the playground, but that does not import build, just compiler.

h, err := hashCompilerBinary()
if err != nil {
panic(err)
}
compilerBinaryHash = h
}

type ImportCError struct {
pkgPath string
}
Expand Down Expand Up @@ -592,17 +611,9 @@ func (s *Session) buildImportPathWithSrcDir(path string, srcDir string) (*Packag
return pkg, archive, nil
}

const (
hashDebug = false
)

var (
gopherJsBinaryHash string
)

func hashGopherJsBinary() (string, error) {
if gopherJsBinaryHash != "" {
return gopherJsBinaryHash, nil
func hashCompilerBinary() (string, error) {
if compilerBinaryHash != "" {
return compilerBinaryHash, nil
}

binHash := sha256.New()
Expand All @@ -618,8 +629,8 @@ func hashGopherJsBinary() (string, error) {
if _, err := io.Copy(binHash, binFile); err != nil {
return "", fmt.Errorf("failed to hash %v: %v", binPath, err)
}
gopherJsBinaryHash = fmt.Sprintf("%#x", binHash.Sum(nil))
return gopherJsBinaryHash, nil
compilerBinaryHash = fmt.Sprintf("%#x", binHash.Sum(nil))
return compilerBinaryHash, nil
}

func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {
Expand All @@ -630,14 +641,14 @@ func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {
// For non-main and test packages we build up a hash that will help
// determine staleness. Set hashDebug to see this in action. The format is:
//
// ## sync
// gopherjs binary hash: 0x519d22c6ab65a950f5b6278e4d65cb75dbd3a7eb1cf16e976a40b9f1febc0446
// build tags:
// import: internal/race
// ## <package>
// compiler binary hash: 0x519d22c6ab65a950f5b6278e4d65cb75dbd3a7eb1cf16e976a40b9f1febc0446
// build tags: <list of build tags>
// import: <import path>
// hash: 0xb966d7680c1c8ca75026f993c153aff0102dc9551f314e5352043187b5f9c9a6
// ...
//
// file: /home/myitcv/gos/src/sync/cond.go
// file: <file path>
// <file contents>
// N bytes
Copy link
Member

@hajimehoshi hajimehoshi Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I feel like this format is a little bit arbitrary. Can we use JSON or YAML or something formatted? I don't have a strong opinion though...

Copy link
Member

@flimzy flimzy Apr 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A possible problem with JSON/YAML is that it's not inherently ordered. While we could find a way to ensure constant ordering, it wouldn't necessarily happen by default, which could lead to false negative cache hits.

Copy link
Member Author

@myitcv myitcv Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal here is to create a hash as quickly as possible in order to determine whether we have a cache miss or not. So yes, whilst the format is arbitrary, it is simple. And requires no additional processing in order to write to the hash. Using JSON/YAML adds more overhead in the middle, overhead that is unnecessary because ultimately the output is a 256 bit value. Humans will only ever read this whilst debugging (which itself will be a rare occurrence) with hashDebug = true.

// ...
Expand All @@ -652,13 +663,7 @@ func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {

if pkg.PkgObj != "" {
fmt.Fprintf(hw, "## %v\n", pkg.ImportPath)

binHash, err := hashGopherJsBinary()
if err != nil {
return nil, err
}

fmt.Fprintf(hw, "gopherjs binary hash: %v\n", binHash)
fmt.Fprintf(hw, "compiler binary hash: %v\n", compilerBinaryHash)

orderedBuildTags := append([]string{}, s.options.BuildTags...)
sort.Strings(orderedBuildTags)
Expand Down