-
-
Notifications
You must be signed in to change notification settings - Fork 7.9k
"Too many open files" in test runs on Python 3.3 #3315
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Labels
Comments
@matthew-brett Is there anything we need to do mpl side about this? |
Sorry - I am not sure of the cause of the problem, or why it is fixed |
mdboom
added a commit
to mdboom/matplotlib
that referenced
this issue
Oct 21, 2015
This should hopefully address the long-reported "Too many open files" error message (Fix matplotlib#3315). To reproduce: On a Mac or Windows box with starvation for file handles (Linux has a much higher file handle limit by default), build the docs, then immediately build again. This will trigger the caching bug. The font cache in the mathtext renderer was broken. It was caching a font file once for every *combination* of font properties, including things like size. Therefore, in a complex math expression containing many different sizes of the same font, the font file was opened once for each of those sizes. Font files are opened and kept open (rather than opened, read, and closed) so that FreeType only needs to load the actual glyphs that are used, rather than the entire font. In an era of cheap memory and fast disk, it probably doesn't matter for our current fonts, but once matplotlib#5214 is merged, we will have larger font files with many more glyphs and this loading time will matter more. The solution here is to do all font file loading in one place and to use `lru_cache` (available since Python 3.2) to do the caching, and to use only the file name and hinting parameters as a cache key. For earlier versions of Python, the functools32 backport package is required. (Or we can discuss whether we want to vendor it).
mdboom
added a commit
to mdboom/matplotlib
that referenced
this issue
Oct 26, 2015
This should hopefully address the long-reported "Too many open files" error message (Fix matplotlib#3315). To reproduce: On a Mac or Windows box with starvation for file handles (Linux has a much higher file handle limit by default), build the docs, then immediately build again. This will trigger the caching bug. The font cache in the mathtext renderer was broken. It was caching a font file once for every *combination* of font properties, including things like size. Therefore, in a complex math expression containing many different sizes of the same font, the font file was opened once for each of those sizes. Font files are opened and kept open (rather than opened, read, and closed) so that FreeType only needs to load the actual glyphs that are used, rather than the entire font. In an era of cheap memory and fast disk, it probably doesn't matter for our current fonts, but once matplotlib#5214 is merged, we will have larger font files with many more glyphs and this loading time will matter more. The solution here is to do all font file loading in one place and to use `lru_cache` (available since Python 3.2) to do the caching, and to use only the file name and hinting parameters as a cache key. For earlier versions of Python, the functools32 backport package is required. (Or we can discuss whether we want to vendor it).
mdboom
added a commit
to mdboom/matplotlib
that referenced
this issue
Oct 27, 2015
This should hopefully address the long-reported "Too many open files" error message (Fix matplotlib#3315). To reproduce: On a Mac or Windows box with starvation for file handles (Linux has a much higher file handle limit by default), build the docs, then immediately build again. This will trigger the caching bug. The font cache in the mathtext renderer was broken. It was caching a font file once for every *combination* of font properties, including things like size. Therefore, in a complex math expression containing many different sizes of the same font, the font file was opened once for each of those sizes. Font files are opened and kept open (rather than opened, read, and closed) so that FreeType only needs to load the actual glyphs that are used, rather than the entire font. In an era of cheap memory and fast disk, it probably doesn't matter for our current fonts, but once matplotlib#5214 is merged, we will have larger font files with many more glyphs and this loading time will matter more. The solution here is to do all font file loading in one place and to use `lru_cache` (available since Python 3.2) to do the caching, and to use only the file name and hinting parameters as a cache key. For earlier versions of Python, the functools32 backport package is required. (Or we can discuss whether we want to vendor it).
mdboom
added a commit
to mdboom/matplotlib
that referenced
this issue
Oct 28, 2015
This should hopefully address the long-reported "Too many open files" error message (Fix matplotlib#3315). To reproduce: On a Mac or Windows box with starvation for file handles (Linux has a much higher file handle limit by default), build the docs, then immediately build again. This will trigger the caching bug. The font cache in the mathtext renderer was broken. It was caching a font file once for every *combination* of font properties, including things like size. Therefore, in a complex math expression containing many different sizes of the same font, the font file was opened once for each of those sizes. Font files are opened and kept open (rather than opened, read, and closed) so that FreeType only needs to load the actual glyphs that are used, rather than the entire font. In an era of cheap memory and fast disk, it probably doesn't matter for our current fonts, but once matplotlib#5214 is merged, we will have larger font files with many more glyphs and this loading time will matter more. The solution here is to do all font file loading in one place and to use `lru_cache` (available since Python 3.2) to do the caching, and to use only the file name and hinting parameters as a cache key. For earlier versions of Python, the functools32 backport package is required. (Or we can discuss whether we want to vendor it).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The OSX testing juggernaut rolls on. For tests on Python 3.3.5, I am getting multiple (> 400) test errors related to "Too many open files":
See:
I wasn't getting this failure for the multiprocess tests on previous Python 3.3 runs (which will have been testing a different commit on the 1.4.x branch):
https://s3.amazonaws.com/archive.travis-ci.org/jobs/30794971/log.txt
I assume this is because the multiprocess tests were each only using a smaller number of open files per process.
Any ideas why this should be happening for Python 3.3 specifically, not 2.7 or 3.4?
The text was updated successfully, but these errors were encountered: