fix: move pubsub publishing out of database transactions to avoid conn exhaustion#17648
Conversation
…n starvation Signed-off-by: Danny Kopping <[email protected]>
6efe39f to
ba2f90a
Compare
Signed-off-by: Danny Kopping <[email protected]>
Signed-off-by: Danny Kopping <[email protected]>
| clock: clock, | ||
| registerer: registerer, | ||
| done: make(chan struct{}, 1), | ||
| provisionNotifyCh: make(chan database.ProvisionerJob, 10), |
There was a problem hiding this comment.
Thumb-suck; I want to protect against temporary network/database blips but also don't want to accumulate too many messages.
…ot affect behaviour Signed-off-by: Danny Kopping <[email protected]>
Signed-off-by: Danny Kopping <[email protected]>
| func (*brokenPublisher) Publish(event string, _ []byte) error { | ||
| // I'm explicitly _not_ checking for EventJobPosted (coderd/database/provisionerjobs/provisionerjobs.go) since that | ||
| // requires too much knowledge of the underlying implementation. | ||
| return xerrors.Errorf("refusing to publish %q", event) |
There was a problem hiding this comment.
Should we fail slowly instead?
e.g. <-time.After(testutil.IntervalFast)
|
@dannykopping is it possible to use another approach, and allow calling func (p *PGPubsub) PublishWithinTx(event string, message []byte, tx database.Store) error {
tx.ExecContext(context.Background(), `select pg_notify(`+pq.QuoteLiteral(event)+`, $1)`, message)
}Implementations which doesn't support txs may implement it: func (p *InMemPubSub) PublishWithinTx(event string, message []byte, tx database.Store) error {
_ = tx // ignored
return p.PublishTx(event, message)
}But it will require some changes in Because we impose restrictions on ourselves, every time we need to use Publish within tx - we have to come up with workaround like in this PR. |
Signed-off-by: Danny Kopping <[email protected]>
@evgeniy-scherbina I'll defer to @spikecurtis about your above question. However, I would imagine that we wouldn't want to overwhelm the API with knowledge of in- or out-of-transaction considerations. There are also some problems around 2PC ("A transaction that has executed NOTIFY cannot be prepared for two-phase commit."), and in general it just makes pubsub a bit less deterministic. |
Database transactions hold onto connections, and
pubsub.Publishtries to acquire a connection of its own. If the latter is called within a transaction, this can lead to connection exhaustion.I plan two follow-ups to this PR:
https://github.com/coder/coder/blob/main/cli/server.go#L2360-L2376
We will then be able to write tests showing how connection exhaustion occurs.
pubsub.Publishfrom being called within a transaction.