Fix online plan bug with multiple serial summers and certain no-ops #524
Conversation
|
https://travis-ci.org/twitter/summingbird/jobs/28356726#L2836 That is not good. |
|
"StormPlatform matches Scala for single step jobs" Was the test that failed. I don't know if this is one of the cases of noisy storm tests. Can anyone comment if we have some reason to think this is a false alarm? |
There was a problem hiding this comment.
any reason to prefer sys.error over the new Exception? is there any difference? (just curious)
There was a problem hiding this comment.
We used sys.error below, and I thought it would be nice to be consistent. If you ask me, I prefer sys.error when you are really signalling a state that can't be handled. I generally like assert, require, and error over throwing the Java exceptions because they cover the three most common cases I see for exceptions.
There was a problem hiding this comment.
is this right? I presume the Set function before was to eliminate dupes?
There was a problem hiding this comment.
The Set looks really dangerous and I think I put it there because Set1 is a very (in terms of memory) Iterable. I'm afraid down-stream of this we are going to get strange effects with chaining. Let's be direct and just use Iterator here.
There was a problem hiding this comment.
do we need to distinct on the key's at that point though?
There was a problem hiding this comment.
also, if we want to eliminate dupes, we should say so in the documentation (and the method name). Eliminating dupes is not part of the mapKeys contract. It is just the keys analog to mapValues.
There was a problem hiding this comment.
Ok thats fine with me if we don't think its in the contract (mapKey's on a map per say would eliminate dupes as part of the operation since it would return a map also).
|
Other than the last question about mapKeys it LGTM when we get a green build. -- We may need a more concise fix if we are to release a bug fix release on the 0.4.x series with it though. |
|
@ianoc what about this is not concise enough? I think there are only two logical changes: 1) the actual bug fix and tests. 2) The removal of the dangerous use of Set, which I can't see any really good reason for. |
|
I'm more than happy to merge to develop, but for back porting into a 0.4.x release we are going with absolute minimums. so if we can do this with a 1 line change that the downstream consumers can vet then I'm for doing that in a patch release branch. |
Fix online plan bug with multiple serial summers and certain no-ops
The comments summarize this one. But the take away is:
We need to look at even more robust tests, and try to think about what a correctness proof for the plans would look like.
There are still many opportunities to make storm plans with fewer bolts. We should take a look at that as we look to squeeze all the perf we can out of storm.