Thanks to visit codestin.com
Credit goes to github.com

Skip to content

compiler: use hash calculation for determining archive staleness #805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
myitcv wants to merge 12 commits into
base: master
Choose a base branch
from
Open

compiler: use hash calculation for determining archive staleness #805

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e807fdb
compiler: use hash calculation for determining staleness
myitcv Apr 23, 2018
e615455
Add --bv flag to test to log verbose build context messages
myitcv Apr 23, 2018
69257f7
Add build tags to the hash
myitcv Apr 23, 2018
f6baa10
More readable version of hash for debug purposes
myitcv Apr 23, 2018
41c38df
Fix file closing
myitcv Apr 23, 2018
d73d4bb
Address feedback from hajimehoshi
myitcv Apr 23, 2018
9c06cba
Refactor to prefer defers for Close
myitcv Apr 23, 2018
82f8b60
Further refactor to use defer for close
myitcv Apr 23, 2018
2e40249
Use better key name for gopherjs binary hash
myitcv Apr 23, 2018
3d5665a
Move compiler hash calc to init phase
myitcv Apr 24, 2018
5c358e3
Remove --bv flag on test
myitcv Apr 24, 2018
1ed4e6a
Add basic test of new hash staleness
myitcv Apr 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address feedback from hajimehoshi
  • Loading branch information
@myitcv
myitcv committed May 4, 2018
commit d73d4bbd142a4758f8d6a02e11b34f5069624cda
72 changes: 57 additions & 15 deletions build/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -592,14 +592,42 @@ func (s *Session) buildImportPathWithSrcDir(path string, srcDir string) (*Packag
return pkg, archive, nil
}

const (
hashDebug = false
)

func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {
if archive, ok := s.Archives[pkg.ImportPath]; ok {
return archive, nil
}

// For non-main and test packages we build up a hash that will help
// determine staleness. Set hashDebug to see this in action. The format is:
//
// ## sync
// gopherjs bin: 0x519d22c6ab65a950f5b6278e4d65cb75dbd3a7eb1cf16e976a40b9f1febc0446
// build tags:
// import: internal/race
// hash: 0xb966d7680c1c8ca75026f993c153aff0102dc9551f314e5352043187b5f9c9a6
// ...
//
// file: /home/myitcv/gos/src/sync/cond.go
// <file contents>
// N bytes
Copy link
Member

@hajimehoshi hajimehoshi Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I feel like this format is a little bit arbitrary. Can we use JSON or YAML or something formatted? I don't have a strong opinion though...

Copy link
Member

@flimzy flimzy Apr 24, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A possible problem with JSON/YAML is that it's not inherently ordered. While we could find a way to ensure constant ordering, it wouldn't necessarily happen by default, which could lead to false negative cache hits.

Copy link
Member Author

@myitcv myitcv Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal here is to create a hash as quickly as possible in order to determine whether we have a cache miss or not. So yes, whilst the format is arbitrary, it is simple. And requires no additional processing in order to write to the hash. Using JSON/YAML adds more overhead in the middle, overhead that is unnecessary because ultimately the output is a 256 bit value. Humans will only ever read this whilst debugging (which itself will be a rare occurrence) with hashDebug = true.

// ...

pkgHash := sha256.New()
var hw io.Writer = pkgHash
var hashDebugOut *bytes.Buffer
Copy link
Member

@hajimehoshi hajimehoshi Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an optional suggestion: Instead of bytes.Buffer, how about using text/template?

Copy link
Member Author

@myitcv myitcv Apr 24, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unclear, why/how would we use text/template here?

if hashDebug {
hashDebugOut = new(bytes.Buffer)
hw = io.MultiWriter(hashDebugOut, pkgHash)
}

if pkg.PkgObj != "" {
fmt.Fprintf(hw, "## %v\n", pkg.ImportPath)

binHash := sha256.New()
binPath, err := os.Executable()
if err != nil {
return nil, fmt.Errorf("could not locate GopherJS binary: %v", err)
Expand All @@ -608,16 +636,16 @@ func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {
if err != nil {
return nil, fmt.Errorf("could not open %v: %v", binPath, err)
}

binHash := sha256.New()
io.Copy(binHash, binFile)
if _, err := io.Copy(binHash, binFile); err != nil {
return nil, fmt.Errorf("failed to hash %v: %v", binPath, err)
}
binFile.Close()
Copy link
Member

@hajimehoshi hajimehoshi Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually (*File).Close doesn't usually cause error (if error happens, this is a critical error), but checking error is not bad here?

Copy link
Member Author

@myitcv myitcv Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd agree; if that fails we're in serious trouble. I think I'll leave this.

fmt.Fprintf(pkgHash, "gopherjs bin: %#x\n", binHash.Sum(nil))
fmt.Fprintf(hw, "gopherjs bin: %#x\n", binHash.Sum(nil))

orderedBuildTags := append([]string{}, s.options.BuildTags...)
sort.Strings(orderedBuildTags)

fmt.Fprintf(pkgHash, "build tags: %v\n", strings.Join(orderedBuildTags, ","))
fmt.Fprintf(hw, "build tags: %v\n", strings.Join(orderedBuildTags, ","))

for _, importedPkgPath := range pkg.Imports {
// Ignore all imports that aren't mentioned in import specs of pkg.
Expand All @@ -644,20 +672,34 @@ func (s *Session) BuildPackage(pkg *PackageData) (*compiler.Archive, error) {
return nil, err
}

fmt.Fprintf(pkgHash, "import: %v\n", importedPkgPath)
fmt.Fprintf(pkgHash, " hash: %#x\n", importedArchive.Hash)
fmt.Fprintf(hw, "import: %v\n", importedPkgPath)
fmt.Fprintf(hw, " hash: %#x\n", importedArchive.Hash)
}

for _, name := range append(pkg.GoFiles, pkg.JSFiles...) {
fp := filepath.Join(pkg.Dir, name)
file, err := s.bctx.OpenFile(fp)
if err != nil {
return nil, fmt.Errorf("failed to open %v: %v", fp, err)
hashFile := func() error {
fp := filepath.Join(pkg.Dir, name)
file, err := s.bctx.OpenFile(fp)
if err != nil {
return fmt.Errorf("failed to open %v: %v", fp, err)
}
defer file.Close()
fmt.Fprintf(hw, "file: %v\n", fp)
n, err := io.Copy(hw, file)
if err != nil {
return fmt.Errorf("failed to hash file contents: %v", err)
}
fmt.Fprintf(hw, "%d bytes\n", n)
return nil
}
fmt.Fprintf(pkgHash, "file: %v\n", fp)
n, _ := io.Copy(pkgHash, file)
file.Close()
fmt.Fprintf(pkgHash, "%d bytes\n", n)

if err := hashFile(); err != nil {
return nil, fmt.Errorf("failed to hash file %v: %v", name, err)
}
}

if hashDebug {
fmt.Printf("%s", hashDebugOut.String())
}

// no commands are archived
Expand Down