-
Notifications
You must be signed in to change notification settings - Fork 881
fix: move pubsub publishing out of database transactions to avoid conn exhaustion #17648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…n starvation Signed-off-by: Danny Kopping <[email protected]>
6efe39f
to
ba2f90a
Compare
Signed-off-by: Danny Kopping <[email protected]>
Signed-off-by: Danny Kopping <[email protected]>
clock: clock, | ||
registerer: registerer, | ||
done: make(chan struct{}, 1), | ||
provisionNotifyCh: make(chan database.ProvisionerJob, 10), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thumb-suck; I want to protect against temporary network/database blips but also don't want to accumulate too many messages.
…ot affect behaviour Signed-off-by: Danny Kopping <[email protected]>
Signed-off-by: Danny Kopping <[email protected]>
func (*brokenPublisher) Publish(event string, _ []byte) error { | ||
// I'm explicitly _not_ checking for EventJobPosted (coderd/database/provisionerjobs/provisionerjobs.go) since that | ||
// requires too much knowledge of the underlying implementation. | ||
return xerrors.Errorf("refusing to publish %q", event) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we fail slowly instead?
e.g. <-time.After(testutil.IntervalFast)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed in 9ddad0b
@dannykopping is it possible to use another approach, and allow calling func (p *PGPubsub) PublishWithinTx(event string, message []byte, tx database.Store) error {
tx.ExecContext(context.Background(), `select pg_notify(`+pq.QuoteLiteral(event)+`, $1)`, message)
} Implementations which doesn't support txs may implement it: func (p *InMemPubSub) PublishWithinTx(event string, message []byte, tx database.Store) error {
_ = tx // ignored
return p.PublishTx(event, message)
} But it will require some changes in Because we impose restrictions on ourselves, every time we need to use Publish within tx - we have to come up with workaround like in this PR. |
Signed-off-by: Danny Kopping <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@evgeniy-scherbina I'll defer to @spikecurtis about your above question. However, I would imagine that we wouldn't want to overwhelm the API with knowledge of in- or out-of-transaction considerations. There are also some problems around 2PC ("A transaction that has executed NOTIFY cannot be prepared for two-phase commit."), and in general it just makes pubsub a bit less deterministic. |
Database transactions hold onto connections, and
pubsub.Publish
tries to acquire a connection of its own. If the latter is called within a transaction, this can lead to connection exhaustion.I plan two follow-ups to this PR:
https://github.com/coder/coder/blob/main/cli/server.go#L2360-L2376
We will then be able to write tests showing how connection exhaustion occurs.
pubsub.Publish
from being called within a transaction.