oy... yeah I know why that is there. This is going to get hairy to fix.

The situation we were trying to fix for here was the whole problem where numpy is a dependency of the setup/build/install process, but it isn't installed ahead of time. So we used a trick to add numpy to the setup_requires entry of setup(), which would download and locally install numpy if numpy isn't already installed in order to have something to build against.

Now, I am trying to remember exactly why the reload is needed, but I remember the whole thing being very fragile in the first place. I am going to need to search my brain for a moment.

373c4d7 looks pretty old. Perhaps the pypi problems it was intended to solve have been fixed. @njsmith Thoughts?

Looking back over that original PR, I see that part of the trick was to implement a sort of "DelayedExtension" class. I am wondering if that might be the reason for the reload? @mdboom might be able to remember.

I should note that it wasn't PyPi that we were trying to deal with, it was the whole "needing numpy for build prior to installing" problem. This was our clever solution that we implemented 3 years ago.

Why do you delete __NUMPY_SETUP__? That is the only thing I can see that requires a reload after import.

That's a pretty complicated setup script:) The whole DelayedExtension class seems unnecessary, nowadays just relying on pip to deal with setup_requires and install_requires should work for getting numpy.get_include() available before starting to build your own extensions.

Also distribute_setup.py is outdated and is better to remove completely.

@rgommers any pointers on docs of how to do this?

It looks like we need the includes to construct the Extension objects to pass into setup() via ext_modules, thus we need to have numpy installed before setup has a chance to call pip to install numpy.

I think the pandas setup.py might have an appropriate incantation.

On Aug 22, 2016 14:47, "Thomas A Caswell" [email protected] wrote:

@rgommers https://github.com/rgommers any pointers on docs of how to do
this?

It looks like we need the includes to construct the Extension objects to
pass into setup() via ext_modules, thus we need to have numpy installed
before setup has a chance to call pip to install numpy.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6928 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAlOaGV3Z35sfCwAhgkhvPFeRiYV9873ks5qihkCgaJpZM4JfW8O
.

pandas seems to do this via numpy_incl = pkg_resources.resource_filename('numpy', 'core/include') and extensive use of the cmdclass machinery.

skimage seems to mostly use from numpy.distutils.misc_util import Configuration, get_numpy_include_dirs and a recursive (?!) setup with setup.py files in all of their sub-packages

scipy seems do do the same recursive setup.py scheme as skimage, but uses np.get_include().

sklearn looks like it has a hard numpy dependency in the setup.py

The key in pandas seems to be

from distutils.command.build_ext import build_ext as _build_ext
class build_ext(_build_ext):
    def build_extensions(self):
        numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')

        for ext in self.extensions:
            if hasattr(ext, 'include_dirs') and not numpy_incl in ext.include_dirs:
                ext.include_dirs.append(numpy_incl)
        _build_ext.build_extensions(self)

which tacks the numpy onto the include_dirs of all of the extensions

and then a

cmd_class['build_ext'] = build_ext

(pandas actually uses a further sub-class of build_ext to but details)

This (presumably) gets run late enough numpy is installed and by using pkg_resource it side-steps some of the 'fully-installed' issues?

any pointers on docs of how to do this?

There are no good docs I'm afraid, just looking at how other packages do this.

The pandas use of pkg_resources.resource_filename('numpy', 'core/include') assumes things about the internal layout of numpy's files, so don't use that. np.get_include() is the public function to use.

I think for large packages the recursive setup.py is much more maintainable than everything in one massive pile, so the way to go. But if you don't want that, it's possible to keep it together. The key thing though is to split code that deals with your external dependencies from code needed to build MPL itself. That way you should always be able to just do np.get_include for your own extensions, without needing complicated distutils command classes or things like DelayedExtension .

I suspect that the path of least code-deformation here is to use the pandas build_ext class instead of our deferred solution.

It is not clear to me if what we currently have, modulo the numpy reload issues, works with just distutils (and not setuptools).

At any rate, this will make it in for 2.0, I am just hoping someone does in before I get to it 😉

"just distutils" is irretrievably broken in general (unfortunately), e.g. you can't properly uninstall packages installed with plain distutils. So requiring setuptools is pretty reasonable these days.

Indeed. Plus that it messes up dev version strings (messes up PEP 440), which is the issue that pushed us over the line for numpy and scipy to start having a hard dependency on setuptools.

numpy has 'fixed' this upstream numpy/numpy#7955

We probably should still fix our setup.py, but it is less urgent

At some point, we will probably have to rework our build system, but considering numpy "fixed" this problem for us, I'll bump this to 2.1.