Codestin Search App

vbatts · 2014-09-15T18:49:37Z

benchmark             old ns/op     new ns/op     delta
BenchmarkTarUntar     5662243       5615513       -0.83%

benchmark             old MB/s     new MB/s     speedup
BenchmarkTarUntar     0.07         0.07         1.00x

benchmark             old allocs     new allocs     delta
BenchmarkTarUntar     10815          10815          +0.00%

benchmark             old bytes     new bytes     delta
BenchmarkTarUntar     520637        520216        -0.08%

ping @unclejack

Signed-off-by: Vincent Batts [email protected]

crosbymichael · 2014-09-15T20:21:33Z

Cross compile is failing

---> Making bundle: cross (in bundles/1.2.0-dev/cross)
Created symlinks: /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/amd64/docker /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/amd64/docker-1.2.0-dev /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/amd64/docker-1.2.0-dev.md5 /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/amd64/docker-1.2.0-dev.sha256
Created binary: /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/386/docker-1.2.0-dev
Created binary: /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/linux/arm/docker-1.2.0-dev
Created binary: /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/darwin/amd64/docker-1.2.0-dev
Created binary: /go/src/github.com/docker/docker/bundles/1.2.0-dev/cross/darwin/386/docker-1.2.0-dev
# github.com/docker/docker/archive
archive/archive.go:194: cannot use stat.Ino (type uint32) as type uint64 in map index
archive/archive.go:198: cannot use stat.Ino (type uint32) as type uint64 in map index

vbatts · 2014-09-16T13:40:57Z

@crosbymichael fixed

PTAL

vbatts · 2014-09-16T14:46:34Z

While this does fix ./archive to handle hardlinks, this does not yet fix layers committed that include hardlinks. http://fpaste.org/133919/10878643/
I am investigating this.

vbatts · 2014-09-19T19:44:05Z

I must have had a funky setup when testing above, because I just tested this on btrfs, dm and aufs and it preserves hardlinks within the same layer http://fpaste.org/134988/15577514/

What will not be preserved is hardlinks to files that a parent layer owns.

vbatts · 2014-09-19T19:44:16Z

@crosbymichael and @unclejack PTAL

unclejack · 2014-09-21T22:58:54Z

@vbatts I was getting failures in the tests with this PR on btrfs. I'll look into it and provide more information.

vbatts · 2014-09-21T23:10:37Z

Ah. Might be worth me adding some integration tests that inodes match
On Sep 21, 2014 6:59 PM, "unclejack" [email protected] wrote:

@vbatts https://github.com/vbatts I was getting failures in the tests
with this PR on btrfs. I'll look into it and provide more information.

—
Reply to this email directly or view it on GitHub
#8046 (comment).

unclejack · 2014-09-22T12:03:53Z

@vbatts This failure I was getting looked like this:

--- FAIL: TestGetContainersExport (0.25 seconds)
        api_test.go:112: The created test file has not been found in the exported image

This is only happening with this PR, master is OK.

vbatts · 2014-09-22T18:37:57Z

@unclejack i've looked into this a bit, and really do not think this failure is a part of this feature, but something racy in the integration tests. If I change the name of the touched file in that test, from "test" to "aaa", then the test passes. It looks like the exported tar stream only gets to around "bin/ifconfig" before it reaches the end of the stream.

tianon · 2014-09-22T23:28:57Z

Ahh, it looks like this is also the fix for docker-library/official-images#184; swell! 👍

vbatts · 2014-09-23T00:10:49Z

Awesome. Hopefully I'll get an integration test done after I get my kids to
bed. Then this could get in for 1.3
On Sep 22, 2014 7:29 PM, "Tianon Gravi" [email protected] wrote:

Ahh, it looks like this is also the fix for
docker-library/official-images#184
docker-library/official-images#184; swell! [image:
👍]

—
Reply to this email directly or view it on GitHub
#8046 (comment).

vbatts · 2014-09-23T14:05:18Z

Rebased and added an integration test PTAL @unclejack

LK4D4 · 2014-09-23T14:39:37Z

@vbatts Integration test failed. See drone details.

vbatts · 2014-09-23T18:30:59Z

Drone failed because of that funky race condition. Again, I'm thinking it's unrelated, but not sure why this expresses it :-\

unclejack · 2014-09-24T00:51:30Z

@vbatts Disabling seenFiles for TarWithOptions fixes the problem:

https://gist.github.com/unclejack/2106d93de3653e07d5e7

The archive ends up getting closed. tw.err = fmt.Errorf("archive/tar: missed writing %d bytes", tw.nb) gets tripped for all subsequent writes. I couldn't find the reason why the archive gets closed.

vbatts · 2014-09-24T01:01:03Z

Oh good. I'll grab that in the morning
On Sep 23, 2014 8:52 PM, "unclejack" [email protected] wrote:

@vbatts https://github.com/vbatts Disabling seenFiles for
TarWithOptions fixes the problem:

https://gist.github.com/unclejack/2106d93de3653e07d5e7

The archive ends up getting closed. tw.err = fmt.Errorf("archive/tar:
missed writing %d bytes", tw.nb) gets tripped for all subsequent writes.
I couldn't find the reason why the archive gets closed.

—
Reply to this email directly or view it on GitHub
#8046 (comment).

vbatts · 2014-09-26T16:04:29Z

Just added a failing test that is more clear than the integration test that is failing. Perhaps i should make a separate PR to get just that unit test and benchmark into master.

Though I'm still not sure why this tar stream seems to end early. :-\

erikh · 2014-09-26T20:03:33Z

why is this code commented out? can we remove it if it's not going to be used?

Ah. I was working to get a unit test to isolate the failure, rather than
going through the integration tests that fails more crypticly. I'll
uncomment it.

On Fri, Sep 26, 2014 at 4:03 PM, Erik Hollensbe [email protected]
wrote:

In archive/archive_test.go:

@@ -63,6 +64,51 @@ func TestCmdStreamGood(t *testing.T) {
}
}

+func TestTarFiles(t *testing.T) {
+

// try without hardlinks

//if err := checkNoChanges(t, 1000, false); err != nil {

//t.Error(err)

//}

// try with hardlinks

why is this code commented out? can we remove it if it's not going to be
used?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8046/files#r18110637.

erikh · 2014-09-26T20:05:07Z

seems legit; LGTM after my comments are addressed.

vbatts · 2014-10-22T20:35:38Z

the test failures i'm seeing are fixed with #8571

vbatts · 2014-10-22T20:37:31Z

also, I hope #8641 would land first, so the benchmarks of this PR are more comparable to master.

vbatts · 2014-10-28T21:07:39Z

@unclejack @tiborvass ALRIGHTY. I found the subtle issue of recording the 0 size for hardlinks, so the writer can keep the position math correct.

benchmark                      old ns/op     new ns/op     delta
BenchmarkTarUntar              3952835       3951407       -0.04%
BenchmarkTarUntarWithLinks     8141363       6601060       -18.92%

benchmark                      old MB/s     new MB/s     speedup
BenchmarkTarUntar              0.10         0.10         1.00x
BenchmarkTarUntarWithLinks     0.05         0.06         1.20x

benchmark                      old allocs     new allocs     delta
BenchmarkTarUntar              7497           7499           +0.03%
BenchmarkTarUntarWithLinks     15011          14868          -0.95%

benchmark                      old bytes     new bytes     delta
BenchmarkTarUntar              356075        356759        +0.19%
BenchmarkTarUntarWithLinks     705642        715747        +1.43%

Also, cleaned up a couple of spots while I was ruling out points-of-concern.

unclejack · 2014-10-28T21:46:08Z

Do we still need this file for something?

I don't guess so, except as a place to have a main() for debugging... I'll remove it if you don't care for it.

That's OK, we can keep it as it is. I was under the impression this served a very particular purpose during the debugging phase.

vbatts · 2014-10-29T02:50:37Z

@unclejack done except for removal of the example app.

LK4D4 · 2014-10-29T16:19:34Z

According to google style guide this ta should be pointer, because it contains pointers.

@LK4D4 are you saying the function needs to be a pointer receiver? (func (ta *tarAppender)...)

vbatts · 2014-10-29T17:58:28Z

@unclejack @LK4D4 @tiborvass rebased with reviews. PTAL.

LK4D4 · 2014-10-29T18:00:33Z

Maybe make sense to create this as pointer too. Just for consistency.

* integration test for preserving hardlinks Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

By default is a demo of file differences, but can be used to create a tar of changes between an old and new path. Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

vbatts · 2014-10-29T18:53:39Z

@LK4D4 fixed. PTAL

LK4D4 · 2014-10-29T19:03:02Z

LGTM

jessfraz · 2014-10-29T19:21:24Z

LGTM

unclejack · 2014-10-29T20:04:17Z

LGTM

archive: preserve hardlinks in Tar and Untar

tonistiigi · 2014-10-29T23:28:12Z

What if stat is not type syscall.Stat_t ?

@tonistiigi point

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 26a96cb to 81044bd Compare September 16, 2014 13:40

tianon mentioned this pull request Sep 22, 2014

Images built using scratch as parent image are bigger docker-library/official-images#184

Closed

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 81044bd to 0e829dd Compare September 23, 2014 14:04

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 0e829dd to 47320e5 Compare September 26, 2014 15:56

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 47320e5 to 14fd587 Compare September 26, 2014 16:05

erikh reviewed Sep 26, 2014
View reviewed changes

vbatts mentioned this pull request Oct 17, 2014

archive: tests and benchmarks for hardlinks #8641

Merged

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 69afc5a to 12cb429 Compare October 22, 2014 20:34

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 12cb429 to 933cfed Compare October 23, 2014 17:11

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from ea4a4aa to 7e0c0f6 Compare October 28, 2014 21:03

unclejack reviewed Oct 28, 2014
View reviewed changes

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 7e0c0f6 to f55e536 Compare October 29, 2014 02:50

LK4D4 reviewed Oct 29, 2014
View reviewed changes

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from f55e536 to 9831c0e Compare October 29, 2014 17:58

LK4D4 reviewed Oct 29, 2014
View reviewed changes

Comment thread pkg/archive/archive.go Outdated

LK4D4 Oct 29, 2014

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make sense to create this as pointer too. Just for consistency.

vbatts and others added 3 commits October 29, 2014 14:52

archive: preserve hardlinks in Tar and Untar

f9f8044

* integration test for preserving hardlinks Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

archive: example app for diffing directories

f710a8d

By default is a demo of file differences, but can be used to create a tar of changes between an old and new path. Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

archive: cleanup and more information

f14a9ed

Signed-off-by: Vincent Batts <[email protected]> Signed-off-by: Vincent Batts <[email protected]>

vbatts force-pushed the vbatts-archive_preserve_hard_links branch from 9831c0e to f14a9ed Compare October 29, 2014 18:53

unclejack added a commit that referenced this pull request Oct 29, 2014

Merge pull request #8046 from vbatts/vbatts-archive_preserve_hard_links

7305716

archive: preserve hardlinks in Tar and Untar

unclejack merged commit 7305716 into moby:master Oct 29, 2014

tonistiigi reviewed Oct 29, 2014
View reviewed changes

fredlf mentioned this pull request Nov 5, 2014

Learning to use Pulls tool, just ignore #8967

Closed

SvenDowideit mentioned this pull request Nov 6, 2014

[DOC] post-1.3.1-docs-update-1 #8991

Closed

ncdc mentioned this pull request Dec 3, 2014

rebase kubernetes openshift/origin#483

Merged

10 tasks

vbatts deleted the vbatts-archive_preserve_hard_links branch April 27, 2016 18:02

Uh oh!

Conversation

vbatts commented Sep 15, 2014

Uh oh!

crosbymichael commented Sep 15, 2014

Uh oh!

vbatts commented Sep 16, 2014

Uh oh!

vbatts commented Sep 16, 2014

Uh oh!

vbatts commented Sep 19, 2014

Uh oh!

vbatts commented Sep 19, 2014

Uh oh!

unclejack commented Sep 21, 2014

Uh oh!

vbatts commented Sep 21, 2014

Uh oh!

unclejack commented Sep 22, 2014

Uh oh!

vbatts commented Sep 22, 2014

Uh oh!

tianon commented Sep 22, 2014

Uh oh!

vbatts commented Sep 23, 2014

Uh oh!

vbatts commented Sep 23, 2014

Uh oh!

LK4D4 commented Sep 23, 2014

Uh oh!

vbatts commented Sep 23, 2014

Uh oh!

unclejack commented Sep 24, 2014

Uh oh!

vbatts commented Sep 24, 2014

Uh oh!

vbatts commented Sep 26, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erikh commented Sep 26, 2014

Uh oh!

vbatts commented Oct 22, 2014

Uh oh!

vbatts commented Oct 22, 2014

Uh oh!

vbatts commented Oct 28, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbatts commented Oct 29, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbatts commented Oct 29, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbatts commented Oct 29, 2014

Uh oh!

LK4D4 commented Oct 29, 2014

Uh oh!

jessfraz commented Oct 29, 2014

Uh oh!

unclejack commented Oct 29, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!