Thanks to visit codestin.com
Credit goes to github.com

Skip to content

bpo-29870: ssl socket leak #981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 9, 2017
Merged

bpo-29870: ssl socket leak #981

merged 1 commit into from
Jun 9, 2017

Conversation

misg
Copy link

@misg misg commented Apr 3, 2017 •
edited
Loading

I'm using Python 3.6 and aiohttp 2.0.5.
Given this server:

import http.server
import ssl

addr = ('localhost', 8081)
httpd = http.server.HTTPServer(addr, http.server.SimpleHTTPRequestHandler)
httpd.socket = ssl.wrap_socket(httpd.socket,
                keyfile='server.key',
                certfile='server.cert',
                server_side=True)
httpd.serve_forever()

and this client (that basically sends get requests to the above server):

import asyncio
import aiohttp
import resource

last = 0

def handler(*_):
        global last
        mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024
        print('Variation:', mem - last)
        last = mem

async def get(session):
        response = await session.request('get', 'https://0.0.0.0:8081') # https (ssl)
        print(response.status)
        response.close()

async def main(loop):
        conn = aiohttp.TCPConnector(verify_ssl=False, enable_cleanup_closed=True)
        session = aiohttp.ClientSession(loop=loop, connector=conn)

        for x in range(500):
                await get(session)
                await asyncio.sleep(1, loop=loop)
                handler()

        session.close()

if __name__ == '__main__':
        loop = asyncio.get_event_loop()
        loop.run_until_complete(main(loop))

I have been able to reproduce the issue. In fact, every 6-7 get requests, I got a Variation: 0.25 printed on my screen that indicates my client uses 0.25 more MB of memory (it just grows indefinitely, I tested it with 10000 requests).

Using ipdb and objgraph led me to this graph where you can clearly see that there are some circular references between SSLProtocol and _SSLPipe objects:

sslprot_backrefs

I know that since PEP 442 it shouldn't be a problem anymore, the python garbage collector doesn't care about the cycle and collect them anyway. But the interesting fact is that breaking this cycle (see my commit) actually fixes the memory leak in my client (no more Variation: 0.25, just always Variation: 0.0). My hypothesis is that circular references between SSLProtocol and _SSLPipe actually hide some circular references between some C objects (that define some tp_dealloc slots), that lead to these memory leaks.

@mention-bot
Copy link

@misg, thanks for your PR! By analyzing the history of the files in this pull request, we identified @1st1, @fafhrd91 and @serhiy-storchaka to be potential reviewers.

@the-knights-who-say-ni
Copy link

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately our records indicate you have not signed the CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

@misg misg mentioned this pull request Apr 3, 2017
Merged
@misg
Copy link
Author

misg commented Apr 3, 2017

(I just signed the CLA today btw)

@misg misg changed the title bpo-29406: asyncio SSL contexts leak sockets after calling close with certain servers bpo-29870: ssl socket leak Apr 10, 2017
@misg
Copy link
Author

misg commented Apr 10, 2017

I digged a little more by putting a printf in every *_dealloc C function (PySSL_dealloc, context_dealloc, memory_bio_dealloc, PySSLSession_dealloc), in the _ssl.c module. Without my patch, none of these printf is actually executed so there is no deallocation at all. With my patch, PySSL_dealloc is executed for each request I do, memory_bio_dealloc is executed twice for each request and context_dealloc is executed once for the entire session.

I think that clearly shows that the above reference cycle hides some reference cycles between these C objects, leading to these leaks.

Honestly, I don't know if my patch is sufficient to patch these leaks. I believe this cycle reference is introduced by the use of a callback after ssl handshake and before ssl shutdown (see attributes handshake_cb and shutdown_cb in _SSLPipe) and I'm not sure breaking this cycle in the _finalize method of SSLProtocol is sufficient since I do not know if this method is truly executed for any request in any case.

@misg
Copy link
Author

misg commented Apr 10, 2017

Note: there is no difference between Python 3.6 and 3.5.2 regarding this issue

@Mariatta Mariatta requested a review from 1st1 June 9, 2017 22:15
@1st1
Copy link
Member

1st1 commented Jun 9, 2017

Thanks for the PR and research, @misg! Will merge this in 3.6.2.

@1st1
Copy link
Member

1st1 commented Jun 9, 2017

@misg I'll merge this PR now, but please submit another PR with some kind of test to avoid regressions in the future.

@1st1 1st1 merged commit d1f5751 into python:master Jun 9, 2017
1st1 added a commit that referenced this pull request Jun 9, 2017
@1st1
Add a NEWS entry for #29870 (PR #981)
71ab858
@misg
Copy link
Author

misg commented Jun 9, 2017

Thanks for your review and merge, @1st1! You're right, I forgot to include some kind of test, I'm going to fix that.

1st1 added a commit that referenced this pull request Jun 9, 2017
@1st1
Add a NEWS entry for #29870 (PR #981) (#2047)
70fede1
1st1 pushed a commit that referenced this pull request Jun 9, 2017
@1st1
Break circular references when closing SSLTransport objects (#981)
dcc7fbc
1st1 pushed a commit that referenced this pull request Jun 9, 2017
@1st1
Break circular references when closing SSLTransport objects (#981)
3d43814
1st1 added a commit that referenced this pull request Jun 9, 2017
@1st1
Break circular references when closing SSLTransport objects (#981) (#…
7f1cd72 
…2048)
1st1 added a commit that referenced this pull request Jun 9, 2017
@1st1
Break circular references when closing SSLTransport objects (#981) (#…
fe9c7a0 
…2049)
@Mariatta
Copy link
Member

Mariatta commented Sep 7, 2017

Does this still need backporting?

@Mariatta
Copy link
Member

Mariatta commented Sep 7, 2017

If not please remove the label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@1st1 1st1 1st1 approved these changes

Assignees

@1st1 1st1

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@misg @mention-bot @the-knights-who-say-ni @1st1 @Mariatta