Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Cannot import matplotlib on Google App Engine dev appserver #1823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mgiuca-google opened this issue Mar 15, 2013 · 6 comments
Closed

Cannot import matplotlib on Google App Engine dev appserver #1823

mgiuca-google opened this issue Mar 15, 2013 · 6 comments

Comments

@mgiuca-google
Copy link
Contributor

Google App Engine is a restricted Python environment which now supports matplotlib on the production server (see Third-party Libraries), due to extensive modifications which Google made to the codebase to get it working. However, the development server runs unmodified code for third-party libraries. The current version of Matplotlib (tested with 9690232 as of 14-03-2013) crashes if "import matplotlib" is run on the dev appserver:

Traceback (most recent call last):
  File "/path/to/matplotlib/__init__.py", line 846, in <module>
    rcParams = rc_params()
  File "/path/to/matplotlib/__init__.py", line 757, in rc_params
    fname = matplotlib_fname()
  File "/path/to/matplotlib/__init__.py", line 646, in matplotlib_fname
    home = get_home()
  File "/path/to/matplotlib/__init__.py", line 300, in wrapper
    ret = func(*args, **kwargs)
  File "/path/to/matplotlib/__init__.py", line 496, in _get_home
    raise RuntimeError('please define environment variable $HOME')
RuntimeError: please define environment variable $HOME

This is just one of the issues (fix that one, and more spring up). There are three major blockers:

  • Matplotlib expects the user to have a writable $HOME directory, so it will not work in any environment where the user does not have a home directory, or where there is a non-writable file system.
  • App Engine does provide a subprocess module, but it is empty. Therefore, any attempt to use subprocess.Popen on App Engine results in AttributeError being raised.
  • The ps backend writes to a temporary file, then reads it back in, even when not using an external distiller. This will not work in any environment where there is a non-writable file system.

We (the Google App Engine Python team) have patched these three issues to get Matplotlib working in our production environment. We propose sending these patches upstream so that Matplotlib will work out-of-the-box on App Engine and other restricted environments which ban file writing and use of subprocess.
@WeatherGod
Copy link
Member

I could have sworn we fixed that first issue (or at least made it configurable) to have a $HOME directory. I recall an issue coming by here where a user simply could not create the font cache file and was willing to sacrifice the performance to get it to work.

The other two issues are also major restrictions. Can you not even write to ram-based file system?

@mgiuca-google mgiuca-google mentioned this issue Mar 15, 2013
Merged
@mgiuca-google
Copy link
Contributor Author

Hmm, well I'm not familiar with the historical codebase, but looking at the current state: the home dir is used in two places: in determining the config dir, and in migrating the old location of matplotlibrc (~/.matplotlibrc) to the config dir. The former can avoid the homedir requirement if you set MPLCONFIGDIR (we could have asked our users to do that, but they wouldn't have any writable place to nominate, so there is still a problem). The latter always requires a home dir; maybe it's new. Anyway, my general fix for the former is to have it fall back to creating a temporary directory, and my fix for the latter is to simply not migrate if there is no home directory. If it cannot create a temporary directory (as on app engine), I carefully avoid creating caches.

There are libraries like pyfilesystem which could provide a RAM-based file system (although I haven't tried it). Though that would introduce a new third-party dependency, along with other questions (Will there be enough RAM? If a server serves for a long time, will it accumulate files in the RAM file system until memory runs out?). There is nothing built into App Engine that provides a RAM-based file system. For the time being, I am simply disabling disk caching if the config dir cannot be created (i.e., MPLCONFIGDIR is not writable AND the user has no writable home dir AND there is no writable temp directory -- all three of which are true on App Engine but should not all be true on any normal environment).

Have a look at Pull Request #1824 and see what you think.

This was referenced Mar 15, 2013
Merged
Merged
@mgiuca-google
Copy link
Contributor Author

Due to the above changes, I have managed to get the HEAD version of Matplotlib to work on the App Engine 1.7.7 dev appserver, without any modifications to Matplotlib! At this time, it requires manual modification of the dev appserver, and is not supported by Google, but it can be made to work.

For any interested parties, the modifications to the dev appserver required are as follows:

  • In _python_runtime.py, find _PYTHON_RUNTIME_PATHS, and add:
    • os.path.join(_DIR_PATH, 'lib', 'dateutil'),
  • In google/appengine/tools/devappserver2/python/sandbox.py:
    • Add ‘matplotlib’ to _C_MODULES.
    • Add 'matplotlib': re.compile(r'matplotlib(\..*)?$'), to NAME_TO_CMODULE_WHITELIST_REGEX.
    • import dateutil
    • In enable_sandbox, add dateutil to modules = [...].

You also need to manually set the backend to Agg. Then you should be able to get a basic graph working. I believe that in some cases you will need to add pytz to the whitelist in the same manner as dateutil is added above.

So this issue is closed from the Matplotlib side. Thanks, everyone, you've been a big help.

@daradib daradib mentioned this issue Mar 3, 2014
Merged
@jonathan-s
Copy link

@mgiuca-google Is the above still valid?

@mgiuca
Copy link

mgiuca commented Mar 2, 2016

I assume so. I haven't been working on App Engine for many years and I don't think this situation has changed. If you find out, please update here.

@pradeepchaudharisc
Copy link

Welcome, if you have reached here, it's 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mgiuca @WeatherGod @jonathan-s @mgiuca-google @pradeepchaudharisc