Denoise Consul namer logging and possibly fix a resource leak #1682
Conversation
|
Further investigation will be necessary in order to determine whether this also resolves any of the resource leaks that have been plaguing the Consul namer for a while. |
There was a problem hiding this comment.
This is a nice fix to the spurious error messages. This may not be an observation leak, though. ServiceRelease is only raised from the closable that also sets stopped=true so after loop is called, execution should stop.
We should be able to confirm this by writing a test that verifies that a mock consul backend does not get polled again after an observation is closed.
| datacenter, | ||
| key.name | ||
| ) | ||
| stopped = true |
There was a problem hiding this comment.
stopped should already be true at this point.
|
@adleong i'm working on that test as we speak. |
As @adleong requested in #1682 (review), I've added a test that we don't observe Consul twice after closing an activity, to see whether or not there's a resource leak here.
|
Okay, update on this: it looks like there's no resource leak here, since I've added a test in ae0662e to assert that Consul isn't polled again after releasing a service. |
## 1.3.1 2017-10-24 * Kubernetes * Fixed a failure to update routing data after restarting watches (#1674). * Ensured that Kubernetes API watch events earlier than the current state are ignored (#1681). * Added support for Istio Mixer precondition checks (#1606). * Removed spurious error message logging from Consul namer (#1682). * Changed DNS SRV record namer to use system DNS resolver configuration (#1679). * Added `timestampHeader` configuration to support New Relic request queue (#1672).
As described in issue #1670, the Consul namer currently logs a scary-looking error message every time we close a watch on a Consul service. This is because the namer's activity handles the
ServiceReleaseexception that's thrown to close a watch that's no longer needed as though it were an error.In addition to logging a spurious error message, if the namer has previously seen a good state from Consul, it handles this "error" by falling back to the last observed good state and restarting the watch, which is, of course, not what we want to do when we've intentionally closed a watch activity that's no longer needed. It's possible that these zombie activities may also cause a resource leak, although I'm not 100% certain this is the case – something else may be closing these activities regardless?
I've modified
SvcAddrto handle theServiceReleaseexception separately from otherThrows, which should stop the spurious error message from being logged and closed watches from being restarted.Fixes #1670