Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Correctly restore connection if MySQL server has gone away #2421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 7, 2019
Merged

Correctly restore connection if MySQL server has gone away #2421

merged 1 commit into from
Oct 7, 2019

Conversation

bramus
Copy link
Contributor

@bramus bramus commented Sep 27, 2019

Based on this code example I have created and deployed a cloud function, which connects over a socket to the Cloud SQL Instance.

I however noticed that it wasn't behaving correctly: after a while I was getting back the Exception pymysql.err.InterfaceError: (0, '') on each request.

Scrolling back in the logs of my cloud function I noticed that the Exception started getting thrown after the connection the MySQL server got lost:

Screenshot 2019-09-26 at 09 41 45

Note this error in the logs, after which the pymysql.err.InterfaceError: (0, '') Exceptions pop up:

pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")

When redeploying the function everything worked fine again, but a day later the same issue occurred again.

I narrowed down the problem to the reconnection logic as copied from the code example. It does not correctly restore the connection when the connection gets lost.

Running the code locally with cloud_sql_proxy, it behaves like so:

Response: OK (e.g. the current time as reported by the MySQL Server)
Response: OK
Response: OK

(manually break pipe by killing cloud_sql_proxy)

Response: pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
Response: pymysql.err.InterfaceError: (0, '')
Response: pymysql.err.InterfaceError: (0, '')
Response: pymysql.err.InterfaceError: (0, '')
Response: pymysql.err.InterfaceError: (0, '')

(restore cloud_sql_proxy connection)

Response: pymysql.err.InterfaceError: (0, '')
Response: pymysql.err.InterfaceError: (0, '')
Response: pymysql.err.InterfaceError: (0, '')

This is a reproduction of the behavior as seen when the function runs on GCP.

With the altered code from this PR, this is the behavior:

Response: OK (e.g. the current time as reported by the MySQL Server)
Response: OK
Response: OK

(manually break pipe by killing cloud_sql_proxy)

Response: pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
Response: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
Response: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
Response: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")
Response: pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 61] Connection refused)")

(manually restore cloud_sql_proxy connection)

Response: OK
Response: OK
Response: OK
Response: OK

As you can see the connection correctly restores after the socket has become available again.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Sep 27, 2019
@andrewferlitsch
Copy link
Contributor

@kurtisvg Ace suggested you look at this PR

Copy link
Contributor

@kurtisvg kurtisvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but in general this logic should be handled by a connection pool.

@gguuss
Copy link
Contributor

gguuss commented Oct 3, 2019

I agree to what kvg said, are we sure this won't increase the billable compute hours for the function by staying warm?

@andrewferlitsch andrewferlitsch merged commit f3fb377 into GoogleCloudPlatform:master Oct 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants