testutils: fix GoroutineAssistant and httputils #1608

steveej · 2015-10-14T11:27:38Z

Closes #1595.
Fixes #1590.

krnowak · 2015-10-14T11:44:48Z

tests/testutils/goroutineassistant.go

I have several issues with GoroutineAssistant's Fatalf:

The s channel should have a buffer (say of two elements), so sending a message there is not blocked.

It should not call Done() on WaitGroup - it is done in every goroutine as a deferred action. The problem was that this deferred action might not be executed, because of being blocked by sending an error to blocking channel.

The return directives were removed after calls to Fatalf in goroutines, so it should imply that Fatalf does not return (it should call runtime.Goexit()).

jonboulle · 2015-10-14T14:11:11Z

Perhaps we need to start thinking about it as a gexpect helper, and use Close during shutdown.

krnowak · 2015-10-14T17:07:03Z

tests/testutils/goroutineassistant.go

Two of three issues I wrote before are still not addressed.

This function is not supposed to return - it should call runtime.Goexit() as the last thing in its body. We assume this function does not return in other places (we removed return clauses).

I think that a.s channel should be non-blocking (as in s: make(chan error, 10)).

Thanks for the reminder. I added the call to runtime.Goexit(). I'm not convinced to make the channel buffered, that would make the code more complex to allow dynamic channel size

I would just make it fixed at 10 and that's it. For now go does not allow to grow channel's buffer. For now we don't use too many goroutines on single assistant (2 at most), so it should be enough in the forseeable future.

Why exactly would you rather have a buffered channel?

@steveej https://github.com/coreos/rkt/pull/1608/files#r42128549

I'd rather have it thought out with unbuffered channel to take advantage of the synchronizing nature of these. This will force us to think about the written test a little more and improve the quality. Buffered channels lead to fire&forget runs, and we'd need another mechanism to synchronize/stop failed tests(routines).

Sending to this channel is usually the last action goroutine does explicitly (that is - not counting the deferred actions).

I wanted to have a buffered channel, just to let the goroutine to die quickly, without waiting for the receiving side of channel, so deferred actions can be executed immediately.

krnowak · 2015-10-15T14:11:33Z

tests/rkt_net_test.go

That will deadlock if spawning fails - it will try to send an error over a blocking channel which is not listened yet by anyone.

Right, we're not supposed to use ga outside of the gouroutines, so SpawnOrFail is not supposed to be used here.

jonboulle · 2015-10-16T01:58:27Z

I tend to agree with krnowak that ultimately this should live in the context, but I think this is OK as a stopgap for now.

steveej · 2015-10-16T09:20:22Z

@krnowak PTAL. rktRunCtx now takes care of registered child processes. It does add a little overhead to kill and wait every child process, but that's a small price to pay for correctness.
I've added one usage example in the TestNetHost* tests and we could use this in all tests that spawn children.

krnowak · 2015-10-16T17:14:20Z

tests/rkt_tests.go

The same thing has to be done also in reset function.

Or maybe put this into runGC? But that might be confusing.

Is it not assumed that ctx.cleanup() is called before ctx.reset()? does it make sense to run reset() without cleanup()? if not, we should have a variable for tracking the context's state and run cleanup() if only reset() is invoked.

krnowak · 2015-10-16T17:25:27Z

The RegisterChild is good for now. Maybe in the follow-up PR we could modify the spawnOrFail function to take the context and register the child immediately.

Also, is the commit "tests/net: one call to ga.Add(1) per goroutine" needed? I suppose we can drop it.

Use a single channel for shutting down GoroutineAssistant, of type error instead of string; in this way, Done() and Fatalf() are serialized and can't race. Fixes #1590

Calling log.Fatal would circumvent proper test shutdown/error propagation. All callers are already checking for the error appropriately.

After a GoroutineAssistant is initialised during tests, all operations that might end in a testing.Fatal should be serialized through it.

The test ACI server closes both Msg and Stop channels on Close(). The Stop channel never receives any messages - it is only used to listen for service shutdown. The channel listening is done in serverHandler, which is run in its own goroutine. The Stop channel was used shortly before, but I wrongly removed it during review process. Because of that, the function never returned and the goroutine was unnecessarily kept alive.

Otherwise there are issues when cleaning up the rkt context.

Since the tests that use the inspect binary rely on stdout parsing, printing this too early will cause a race condition between the serve and test code. Simply moving the print to a point after the Listen() call prevents that.

* don't branch since Fatalf will not return * exit the calling goroutine in Fatalf

When tests spawn children and then cause a failure, they don't have the chance to wait for the children to complete. These tests are supposed to use the ctx.RegisterChild function for every child so that ctx.cleanup() will be able to handle child shutdown.

krnowak · 2015-10-16T17:48:50Z

LFAD if green.

jonboulle · 2015-10-16T18:01:02Z

tests/rkt_tests.go

What about the errors from these two?

We could pass it on and then handle (ignore) it in the invoking function, would you prefer that?

I suggest doing a nominal check and just logging for information purposes. I'm just trying to future-proof us against the kind of murky mad situation that got us here in the first place.

If you feel strongly against that, let's at least move to the better practice/style of explicitly ignoring the error:
_ = child.Cmd.Process.Wait()

I'm fine with logging the error. If it was just me, we'd be logging a lot more.

jonboulle · 2015-10-16T21:57:54Z

@steveej https://www.youtube.com/watch?v=AR5FEQnmOcc

krnowak · 2015-10-19T12:28:05Z

This PR is dragging on and on and is gathering more and more changes. I'd like to merge ASAP, but first some of my notes/questions first:

Calling t.Fatal itself does not kill the "child" goroutines, only the goroutine that called it. They can outlive the goroutine that created them easily (actually, my commit "functional tests: Return from a goroutine when the ACI server closes" is a kind of a proof of it - I decided to fix that when I noticed the serverHandler goroutine in stacktrace resulting from some panic in a completely unrelated test). And to be clear - I'm talking about T.Fatal, not GoroutineAssistant.Fatal.
- GoroutineAssistant.Fatal also kills a goroutine it called, but it may get blocked by sending a message to a channel with no receiver. Which leads to next issue.
Do we want to require that all goroutines created by the test should be killed together with the test? I am strongly in favor of this.
We have some prior art with tackling the logging issue - it is in the ACI test server. I, instead of providing a logging interface, just provided two channels - one for getting logs and one for letting know when the service is closed. Today, I would say that just one channel is enough (when it's closed, the service stopped). The code:
- runServer and serverHandler in auth tests: https://github.com/steveeJ/rkt/blob/6a22db07df544d127b3e955ec776e1e71cd45cd3/tests/rkt_auth_test.go#L145
- serverHandler (the other one that the one in previous point) in test aci server: https://github.com/steveeJ/rkt/blob/6a22db07df544d127b3e955ec776e1e71cd45cd3/tests/test-auth-server/aci/server.go#L45
- Server and NewServer in test aci server: https://github.com/steveeJ/rkt/blob/6a22db07df544d127b3e955ec776e1e71cd45cd3/tests/test-auth-server/aci/server.go#L159

But can we please merge it already? It fixes those pesky failures in networking tests and we are good. I'd prefer to address the other issues (no leftover goroutines, agree on logging solution, proper rkt registration for all tests and so on) in separate, uh, issues. Github issues.

jonboulle · 2015-10-19T17:46:30Z

SGTM

testutils: fix GoroutineAssistant and httputils

krnowak reviewed Oct 14, 2015
View reviewed changes

steveej added kind/bug area/testing labels Oct 14, 2015

steveej self-assigned this Oct 14, 2015

steveej added this to the v0.10.0 milestone Oct 14, 2015

krnowak reviewed Oct 15, 2015
View reviewed changes

krnowak mentioned this pull request Oct 15, 2015

testutils: fix GoroutineAssistant and httputils #1595

Closed

jonboulle added needs/some-changes priority/P1 labels Oct 16, 2015

steveej changed the title ~~testutils: fix GoroutineAssistant and httputils #1595~~ testutils: fix GoroutineAssistant and httputils Oct 16, 2015

steveej mentioned this pull request Oct 16, 2015

functional tests: Return from a goroutine when the ACI server closes #1545

Closed

krnowak reviewed Oct 16, 2015
View reviewed changes

jonboulle and others added 8 commits October 16, 2015 19:33

tests: fix race in GoroutineAssistant

0ff4331

Use a single channel for shutting down GoroutineAssistant, of type error instead of string; in this way, Done() and Fatalf() are serialized and can't race. Fixes #1590

testutils: properly return error from HttpGet

7255852

Calling log.Fatal would circumvent proper test shutdown/error propagation. All callers are already checking for the error appropriately.

tests: fix waitgroup initialization in net tests

586b321

tests: use GoroutineAssistant for all operations

93df7f4

After a GoroutineAssistant is initialised during tests, all operations that might end in a testing.Fatal should be serialized through it.

tests: do not error on IsNotExist during cleanup

f56e4a4

tests: wait for other goroutines on teardown

d385f9b

Otherwise there are issues when cleaning up the rkt context.

tests: fix print logic after HTTP get

2efce15

steveej added 5 commits October 16, 2015 19:33

testutils/goroutineassistant: fail on first error

01634f7

goroutineassistant: fix Fatalf logic

183288a

* don't branch since Fatalf will not return * exit the calling goroutine in Fatalf

TestNetHost*: register children with context

8d40335

steveej added needs/review and removed needs/some-changes labels Oct 16, 2015

krnowak added reviewed/lgtm and removed needs/review labels Oct 16, 2015

jonboulle reviewed Oct 16, 2015
View reviewed changes

steveej added needs/review and removed reviewed/lgtm labels Oct 16, 2015

steveej added 2 commits October 17, 2015 00:04

testsutils logger: add wrapper interface/type for logging in tests

c35ec98

tests net related: improve logging by using the new utility package

f0e19cf

steveej mentioned this pull request Oct 19, 2015

functional tests QA #1633

Open

4 tasks

steveej added a commit that referenced this pull request Oct 19, 2015

Merge pull request #1608 from steveeJ/goassistant

1d21d76

testutils: fix GoroutineAssistant and httputils

steveej merged commit 1d21d76 into rkt:master Oct 19, 2015

steveej deleted the goassistant branch October 19, 2015 19:49

Uh oh!

testutils: fix GoroutineAssistant and httputils #1608

testutils: fix GoroutineAssistant and httputils #1608

Uh oh!

Conversation

steveej commented Oct 14, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle commented Oct 14, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle commented Oct 16, 2015

Uh oh!

steveej commented Oct 16, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

krnowak commented Oct 16, 2015

Uh oh!

krnowak commented Oct 16, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle commented Oct 16, 2015

Uh oh!

krnowak commented Oct 19, 2015

Uh oh!

jonboulle commented Oct 19, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants