I think both your questions may be related #354. I think miniconda is using setuptools instead of distutils(can you confirm this?) and setuptools is putting jep in an egg instead of where we expect it. Since distutils is deprecated I think sysconfig would be better than distutils.sysconfig but since we have other work to do to move off distutils it doesn't really matter which you use here.

On a python install using distutils I do not need to do install_lib, so I am not sure if it needs to be in the documentation. We would need to test it in more environment and make sure it works. I would really like to get some resolution for #354 before releasing jep 4.1 so we might not care what distutils does since you are targeting 4.1 and we will likely need to update many things for that transition.

We do not have a firm release schedule but I am not planning to release jep 4.1 before October. If you need this in sooner we could consider putting this in 4.0.4, let me know if that matters to you.

@bsteffensmeier PR updated to use the basename of the detected path from get_libpython(). A question remains about what to do if that detected path is the multiarch one, since that nuance would be lost when setting PYTHON_LDLIBRARY. Maybe this is fine though? I don't know enough about how this stuff works to have any opinion.

Also updated the setup.py logic to fall back to LDLIBRARY again if get_libpython() fails to find anything real.

We do not have a firm release schedule but I am not planning to release jep 4.1 before October.

On my side, I think an October release would be OK, although of course releasing earlier helps the Linux portion of our user community to try out this functionality more easily. I don't want to make extra work for you, but I'm happy to rebase this PR against dev_4.0 if releasing 4.0.4 would be a simple matter for you.

miniconda is using setuptools instead of distutils(can you confirm this?)

I guess so, since this is the output I see:

$ python setup.py build install install_lib
numpy not found, running without numpy support
running build
running setup_java
running build_java
running build_jar
running build_py
running build_ext
running build_scripts
/home/curtis/miniconda3/envs/jeptest/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running install
/home/curtis/miniconda3/envs/jeptest/lib/python3.8/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
...

Which looks a lot like setuptools to me. 😆

The setup.py script contains the line from distutils.core import setup, Extension and then a direct call to setup(...) in the main block, but I guess setuptools now adopted distutils and I definitely see the _distutils_hack in my environment, so, yeah.

Ah, right. But then... how does MULTIARCH support work?

PYTHON_LDLIBRARY is used to reload a library that is already loaded. It's my understanding that at that point the path to the library is irrelevant and as long as the basename matches it will be reloaded so what you have is correct.

Also, do you think get_libpython should call get_python_lib_dir instead of going straight for LIBDIR?

I am fine either way. It looks like get_python_lib_dir is dead code now and only special on windows while this code is only relevant on linux so it really makes no difference right now.

On my side, I think an October release would be OK, although of course releasing earlier helps the Linux portion of our user community to try out this functionality more easily. I don't want to make extra work for you, but I'm happy to rebase this PR against dev_4.0 if releasing 4.0.4 would be a simple matter for you.

Let's leave it in dev_4.1. If anything else comes up for a new 4.0 release I will definitely cherry pick it back but otherwise I would rather wait for 4.1.

The python code looks good to me but I do not understand the change to the README. I do not see any difference in running setup.py with or without install_lib, as far as I can tell install always runs the install_lib task. The jep script has problems for me in miniconda because of setuptools putting the libraries in an egg. From my brief research I am thinking the readme should probably recommend running pip install . for a source install. See https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg

Thanks @bsteffensmeier. I removed the README change for now, until I have time to investigate in more detail why it was necessary for me. If I figure anything out related to that, I'll open a new issue or PR for it.