this should no longer be needed. This was added to Algebird as the default for tuples.

why is this better than the default Semigroup?

this is still unclear to me.

So, from the above, it seems that we are using this number to be the max number of keys in a cache on the flatmappers, and then using the fact that Maps can be summed, and then emitted as a single batch.

Can you explain more the whole algorithm here?

I would prefer if we could just get the effect of batching here. For instance, the cache emits blocks of results, those could be written as a List, and sent over (perhaps with a max size and chunked).

So we can't just emit the cache as a list of keys, we need an outer key grouping. Since we are on a key grouped level, we need to have a stable mapping of K -> OuterK. Here we are configuring a setting of how big the space we should map the original K into, a multiple of the available consumers. Generic batching still wouldn't really solve this I don't think?

shouldn't this go in AllOpts and get a comment? I really don't know what this does.

Its not there because its not a user set variable, not really sure where it should go. I didn't want to pass around an integer. It contains the size space which to mod the hash of the key into. (calculated in storm platform)

Should it be private in that case?

Yep, will change

why wrapping S here rather than leave it opaque? Seems like the previous PR did the opposite: move from (Timestamp, T) => T, here we are taking (what I think) is an opaque parameter S and putting structure we don't use. Why not just instantiante with S=InputState[S1] for some S1?

I use the InputState portion further down in the Summer class now for fan out. I could put S in the type parameter of Summer to be S <: InputState[_] ?

ahh... here is where you use the concrete nature of InputState. This is new right?.

Yep exactly. S <: InputState[_] compiles nicely though, so we could use it?

BTW: This seems super easy to get wrong. fanOut(size) seems more intuitive.

Also, state.fanOut can throw, but that will cause an exception, not return a Future.exception. What about:

try {
  // code you have above here
}
catch {
  case t: Throwable => Future.exception(t)
}

why is 1 not enough here? Can you add a comment.

This really came from Aaron, i think some sort of notion that incase the space gets lob sided in some manner. I can't really see why it shouldn't be one if everything is well behaved.

size > 0 or we would not have recieved anything, right? Can we put:

assert(size > 0, "Input maps must be non-empty")

else, we are not going to do the acking correctly, right?

.nonEmpty