enable heartbeat (by default) for queueconsumer#383
Conversation
davidszotten
left a comment
There was a problem hiding this comment.
looks like a bunch of failures on travis. :(
| def consume(self, limit=None, timeout=None, safety_interval=0.1, **kwargs): | ||
| """ Lifted from Kombu. | ||
|
|
||
| We switch the order of the `break` and `self.on_iteration()` to |
There was a problem hiding this comment.
is this no longer an issue? how long is that timeout and would stopping have to wait for it without this?
There was a problem hiding this comment.
The default timeout is 0.1 seconds so I don't think there's anything to worry about there. If timeout is never set as per the comment it has no effect anyway.
@kollhof can you remember why you did this?
| heartbeat = self.container.config.get( | ||
| HEARTBEAT_CONFIG_KEY, DEFAULT_HEARTBEAT | ||
| ) | ||
| self._connection = Connection(self.amqp_uri, heartbeat=heartbeat) |
There was a problem hiding this comment.
is there any overhead to heartbeats?
There was a problem hiding this comment.
There is some extra network traffic but no performance implication as far as I'm aware. Services will probably get faster because broken connections are discovered earlier.
|
|
||
| if self._connection is None: | ||
| self._connection = Connection(self.amqp_uri) | ||
| heartbeat = self.container.config.get( |
There was a problem hiding this comment.
are these just ignored with older (or non-forked) amqp?
There was a problem hiding this comment.
If this works with our oldest supported kombu then I think we're fine. It's ignored for non-pyamqp transports and has been supported there since 0.9.1.
There was a problem hiding this comment.
ignored (as opposed to explodes) is great
|
The test suite passed for me locally (OS X, Python 3.4.3) but the failures here do seem to be real. I will investigate. |
| # proxy.task() will hang forever because it generates an error | ||
| proxy.task.call_async() | ||
|
|
||
| with eventlet.Timeout(1): |
There was a problem hiding this comment.
this was taking longer than a second?
did we make stuff slower or is this travis having issues?
There was a problem hiding this comment.
Hard to say. I added a timer context manager around this locally and container.wait only ever took ~0.005 seconds.
There was a problem hiding this comment.
But I also can't get it to fail locally...
There was a problem hiding this comment.
hm. looking at travis, builds historically take ~3 minutes. this pr is around 9. worth checking if this is travis having issues or this pr
There was a problem hiding this comment.
Travis is variable. Re-running master now is taking 4-5 mins: https://travis-ci.org/nameko/nameko/builds/177243811
There is some small impact here on tear-down times, but I think it is just 0.1 seconds. It becomes apparent in here because we do it a lot.
Do we really care that services take slightly longer to stop? I could retrofit the call to conn.heartbeat_check() into our lifted method, but I'd rather remove the override. We wouldn't justify adding the override for a small improvement in test-run time.
There was a problem hiding this comment.
should we add these back? (make the failure case a timeout instead of a hung test)
i guess they also helped us catch the performance issue
There was a problem hiding this comment.
I don't think we should put them back. We use pytest-timeout to catch hanging tests now, which is what these were originally used for.
Tracking the average test execution time would be a much better way to spot performance issues. It probably wouldn't be too difficult to send stats to a free tier on something like https://plot.ly.
There was a problem hiding this comment.
ah, forgot about pytest-timeout. though i can't see where we switch it on..
There was a problem hiding this comment.
Ha, I thought it defaulted to a 30 second timeout if installed. But apparently not. I'll add it
There was a problem hiding this comment.
sounds a bit long for an individual test but we can tweak later
|
On reflection I think the best resolution for now is to restore the I have changed the I'd like to land this PR as it is so we get a) "default" timeout parameters and b) consumer heartbeats. We can assess whether there's a better way to use the |
|
|
||
| return self._connection | ||
|
|
||
| def consume(self, limit=None, timeout=None, safety_interval=1, **kwargs): |
There was a problem hiding this comment.
did you move this intentionally? makes the diff harder to see
There was a problem hiding this comment.
Sorry, not intentional. Have moved it back again
|
i guess once the toxiproxy lands we can add integration tests for the heartbeat |
|
👍 needs changelog |
Kombu does actually support consumer heartbeats in more recent versions, if used with py-amqp.
This PR also removes an override that was lifted (in 2013) from Kombu for (I think) reasons of optimisation. The test suite certainly passes without it, so I think it can safely be dropped.