Stop k8s watches from time travelling #1681
Conversation
| // Ignore any cases where we receive a resource version lower | ||
| // than the last received resource version, by creating a sliding | ||
| // two-item window over the stream of watch events. | ||
| stream.paired |
There was a problem hiding this comment.
I don't think this works. Consider:
stream = [3, 1, 2]
paired = [(x,3), (3,1), (1,2)]
filtered = [(x,3), (1,2)]
mapped = [3,2]
I think you could probably accomplish this with a scanLeft...
stream.scanLeft[(W, Boolean)]((Zero, false)) { case ((largest, incr), w) =>
if (w > largest) (w, true)
else (largest, false)
}.filter(_._2).map(_._1)
or something like that
There was a problem hiding this comment.
Yeah, you're right; 0208ec4 should be more correct.
| AsyncStream.fromFuture(f) | ||
| } | ||
| } | ||
| // Ignore any cases where we receive a resource version lower |
There was a problem hiding this comment.
defining this as a method def increasingSubsequence[T <: Ordered[T]](stream: AsyncStream[T]): AsyncStream[T] would make the logic of this method easy to test (since you can just test it on an AsyncStream of numbers).
| trait Watch[O <: KubeObject] extends Ordered[Watch[O]] { | ||
| def resourceVersion: Option[String] | ||
|
|
||
| private[Watch] def versionNum: Option[Long] = for { |
There was a problem hiding this comment.
what is the visibility scope here? Is this just equivalent to private?
There was a problem hiding this comment.
yeah, i just thought that – since Scala's visibility modifiers can be confusing – it was clearer to make the scope explicit?
## 1.3.1 2017-10-24 * Kubernetes * Fixed a failure to update routing data after restarting watches (#1674). * Ensured that Kubernetes API watch events earlier than the current state are ignored (#1681). * Added support for Istio Mixer precondition checks (#1606). * Removed spurious error message logging from Consul namer (#1682). * Changed DNS SRV record namer to use system DNS resolver configuration (#1679). * Added `timestampHeader` configuration to support New Relic request queue (#1672).
The success rate mini chart shows a colour based on SR, and also shows the SR via the proportion of the chart that's filled out. If the success rate is 0% (as in the VotePoop endpoint in the emojivoto demo), the chart would be zero percent filled out, causing it to be entirely gray. Really, it should be entirely red, since zero SR is pretty bad. Fix: fully fill the bar with red if there is a zero SR
I've modified our code for watching Kubernetes resources so that it will ignore events whose resource versions are older than the resource version of the previous event. We do this by creating a two-item sliding window over the stream of watch events, and filtering out those tuples where the first item (the previous event) is older than the second item (the received event). This should ensure that we never travel back in time when the Kubernetes API is out of sync.
I've added a test for
EndpointsNamerandServiceNamerthat assert that the namer's state does not update when receiving events in reverse order. Furthermore, a number of existing tests had to be modified to ensure all resource versions were sequential – we probably had been lazy in the past about copying and pasting JSON strings in tests without updating their resource versions.Closes #1680