Fix install from sdist and bdist_wheel #27

mattrobenolt · 2015-10-11T11:08:06Z

Ok, so this one fixes up lots of things. (not sure how this was working before tbh)

python setup.py install didn't work on on sdist artifact because it was attempting to run install_regexes which don't exist. They're expected to already be in the bundled artifact. This would affect installing ua-parser from PyPI for example since install gets run during pip install.
After an sdist run, we don't need the yaml file anymore, since we have "compiled" into a json file. Meaning, we can leave the yaml file behind, and only package up the json file. This also means that we can remove pyyaml as a runtime dependency, and only keep it as a setup dependency.
Add support for compiling a wheel with bdist_wheel.

So here's the blobs of data that I tested with:

Output from compiling the sdist and wheel:

$ python setup.py sdist bdist_wheel
running sdist
Copying regexes.yaml to package directory...
Converting regexes.yaml to regexes.json...
running egg_info
creating ua_parser.egg-info
writing ua_parser.egg-info/PKG-INFO
writing top-level names to ua_parser.egg-info/top_level.txt
writing dependency_links to ua_parser.egg-info/dependency_links.txt
writing pbr to ua_parser.egg-info/pbr.json
writing manifest file 'ua_parser.egg-info/SOURCES.txt'
reading manifest file 'ua_parser.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ua_parser.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
creating ua-parser-0.5.0
creating ua-parser-0.5.0/ua_parser
creating ua-parser-0.5.0/ua_parser.egg-info
making hard links in ua-parser-0.5.0...
hard linking MANIFEST.in -> ua-parser-0.5.0
hard linking README.md -> ua-parser-0.5.0
hard linking setup.cfg -> ua-parser-0.5.0
hard linking setup.py -> ua-parser-0.5.0
hard linking ./ua_parser/__init__.py -> ua-parser-0.5.0/./ua_parser
hard linking ./ua_parser/user_agent_parser.py -> ua-parser-0.5.0/./ua_parser
hard linking ./ua_parser/user_agent_parser_test.py -> ua-parser-0.5.0/./ua_parser
hard linking ua_parser/regexes.json -> ua-parser-0.5.0/ua_parser
hard linking ua_parser.egg-info/PKG-INFO -> ua-parser-0.5.0/ua_parser.egg-info
hard linking ua_parser.egg-info/SOURCES.txt -> ua-parser-0.5.0/ua_parser.egg-info
hard linking ua_parser.egg-info/dependency_links.txt -> ua-parser-0.5.0/ua_parser.egg-info
hard linking ua_parser.egg-info/not-zip-safe -> ua-parser-0.5.0/ua_parser.egg-info
hard linking ua_parser.egg-info/pbr.json -> ua-parser-0.5.0/ua_parser.egg-info
hard linking ua_parser.egg-info/top_level.txt -> ua-parser-0.5.0/ua_parser.egg-info
copying setup.cfg -> ua-parser-0.5.0
Writing ua-parser-0.5.0/setup.cfg
creating dist
Creating tar archive
removing 'ua-parser-0.5.0' (and everything under it)
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/ua_parser
copying ./ua_parser/__init__.py -> build/lib/ua_parser
copying ./ua_parser/user_agent_parser.py -> build/lib/ua_parser
copying ./ua_parser/user_agent_parser_test.py -> build/lib/ua_parser
copying ./ua_parser/regexes.json -> build/lib/ua_parser
installing to build/bdist.macosx-10.11-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.11-x86_64
creating build/bdist.macosx-10.11-x86_64/wheel
creating build/bdist.macosx-10.11-x86_64/wheel/ua_parser
copying build/lib/ua_parser/__init__.py -> build/bdist.macosx-10.11-x86_64/wheel/ua_parser
copying build/lib/ua_parser/regexes.json -> build/bdist.macosx-10.11-x86_64/wheel/ua_parser
copying build/lib/ua_parser/user_agent_parser.py -> build/bdist.macosx-10.11-x86_64/wheel/ua_parser
copying build/lib/ua_parser/user_agent_parser_test.py -> build/bdist.macosx-10.11-x86_64/wheel/ua_parser
running install_egg_info
Copying ua_parser.egg-info to build/bdist.macosx-10.11-x86_64/wheel/ua_parser-0.5.0-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.11-x86_64/wheel/ua_parser-0.5.0.dist-info/WHEEL

Our dist output has both the tar.gz and whl files.

$ ls dist
ua-parser-0.5.0.tar.gz               ua_parser-0.5.0-py2.py3-none-any.whl

Listing contents of sdist shows the regexes.json packaged inside:

$ tar ztvf dist/*.tar.gz
drwxr-xr-x  0 matt   staff       0 Oct 11 04:02 ua-parser-0.5.0/
-rw-r--r--  0 matt   staff      49 Oct 11 03:50 ua-parser-0.5.0/MANIFEST.in
-rw-r--r--  0 matt   staff    1142 Oct 11 04:02 ua-parser-0.5.0/PKG-INFO
-rw-r--r--  0 matt   staff    2684 Oct 11 02:33 ua-parser-0.5.0/README.md
-rw-r--r--  0 matt   staff      88 Oct 11 04:02 ua-parser-0.5.0/setup.cfg
-rw-r--r--  0 matt   staff    2426 Oct 11 03:49 ua-parser-0.5.0/setup.py
drwxr-xr-x  0 matt   staff       0 Oct 11 04:02 ua-parser-0.5.0/ua_parser/
-rw-r--r--  0 matt   staff      20 Oct 11 02:33 ua-parser-0.5.0/ua_parser/__init__.py
-rw-r--r--  0 matt   staff  117000 Oct 11 04:02 ua-parser-0.5.0/ua_parser/regexes.json
-rw-r--r--  0 matt   staff   18685 Oct 11 03:38 ua-parser-0.5.0/ua_parser/user_agent_parser.py
-rw-r--r--  0 matt   staff    9685 Oct 11 02:44 ua-parser-0.5.0/ua_parser/user_agent_parser_test.py
drwxr-xr-x  0 matt   staff       0 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/
-rw-r--r--  0 matt   staff       1 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/dependency_links.txt
-rw-r--r--  0 matt   staff       1 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/not-zip-safe
-rw-r--r--  0 matt   staff      47 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/pbr.json
-rw-r--r--  0 matt   staff    1142 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/PKG-INFO
-rw-r--r--  0 matt   staff     350 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/SOURCES.txt
-rw-r--r--  0 matt   staff      10 Oct 11 04:02 ua-parser-0.5.0/ua_parser.egg-info/top_level.txt

And also exists inside the whl.

$ tar ztvf dist/*.whl
-rwxrwxrwx  0 0      0          20 Oct 11 02:33 ua_parser/__init__.py
-rwxrwxrwx  0 0      0      117000 Oct 11 04:02 ua_parser/regexes.json
-rwxrwxrwx  0 0      0       18685 Oct 11 03:38 ua_parser/user_agent_parser.py
-rwxrwxrwx  0 0      0        9685 Oct 11 02:44 ua_parser/user_agent_parser_test.py
-rwxrwxrwx  0 0      0          10 Oct 11 04:02 ua_parser-0.5.0.dist-info/DESCRIPTION.rst
-rwxrwxrwx  0 0      0        1172 Oct 11 04:02 ua_parser-0.5.0.dist-info/metadata.json
-rwxrwxrwx  0 0      0          47 Oct 11 04:02 ua_parser-0.5.0.dist-info/pbr.json
-rwxrwxrwx  0 0      0          10 Oct 11 04:02 ua_parser-0.5.0.dist-info/top_level.txt
-rwxrwxrwx  0 0      0         110 Oct 11 04:02 ua_parser-0.5.0.dist-info/WHEEL
-rwxrwxrwx  0 0      0        1132 Oct 11 04:02 ua_parser-0.5.0.dist-info/METADATA
-rwxrwxrwx  0 0      0         936 Oct 11 04:02 ua_parser-0.5.0.dist-info/RECORD

Now, if we take the dist/ folder, we can install the files individually with pip without any issues. ala pip install dist/ua-parser-0.5.0.tar.gz or pip install dist/ua_parser-0.5.0-py2.py3-none-any.whl

mattrobenolt · 2015-10-11T11:09:29Z

setup.py

    setup_requires=['pyyaml'],
-    install_requires=['pyyaml'],
+    install_requires=[],


plz let me know if you feel strongly against this change since it'd potentially break if someone had UA_PARSER_YAML set.

Personally, I'd rather catch the ImportError, and message at that point.

mattrobenolt · 2015-10-11T11:37:48Z

I think this will fix #14 too.

mattrobenolt · 2015-10-11T11:41:14Z

Oh, I see this is also directly conflicting with #12.

/cc @crbunney

better bot detection

Fix install from sdist and bdist_wheel

elsigh · 2015-10-15T16:27:56Z

Sorry for not merging this sooner - it sounds like you put in a lot of effort and it's green ;)

crbunney · 2015-10-15T16:35:14Z

setup.py

@@ -38,11 +35,6 @@ def run(self):
        _sdist.run(self)


-class install(_install):


I'm a late to the party, I know, but removing this essentially undoes #12, was there a particular reason for it being removed or can it be reinstated?

Yeah, this is why I tried pinging you in #27 (comment)

So install step shouldn't be doing this work. Install is what gets run over an sdist bundle.

Stepping back, if we were to publish to pypi as we've been doing for years, we'd run setup.py sdist upload, which will create the tar.gz, and upload to pypi. Now, if we want to install, we can run pip install ./dist/whatever.tar.gz to install directly from the tarball.

This effectively extracts the tarball, and runs setup.py install. The problem is that we can't call install_regexes at this point since it's not a source. install already expects the artifacts to be bundled up from a previous run of sdist.

I'm not convinced that we can actually support your use case without breaking the 90% use case of installing from pypi.

Unless we make install fail gracefully. That way it would work if the artifacts exist, and noop, or run install_regexes if they don't, for your case of trying to install directly from GitHub.

Does that make sense?

fwiw, this is only complicated because of these weird submodule dependency on uap-core. :)

Yes, I saw your comment, I'm afraid I've only just picked it up, sorry about that.

I didn't realise that my PR broke the pypi packaging, so I can immediately understand why you removed it.

What you suggest does make sense :)

I'll figure it out and send new PR to address the install issue since I understand your use case now.

Though, frankly, I'd say that if you're installing from GitHub, should always use -e IMO. ;)

But I'll see what I can do.

Unfortunately we're installing via a setup.py and dependency_links - I've not found a way of specifying the equivalent of -e in that situation.

i.e.

<snip> dependency_links=['git+https://github.com/crbunney/uap-python.git@fix_pip_install_from_source#egg=ua-parser-0.4.0'], # Run pip install with --process-dependency-links flag to use dependency_links install_requires=['ua-parser==0.4.0'], <snip>

because we wanted to use some code that wasn't yet on pypi.

Hence the unusual use case

Ah, alright. I'll take that into account when I poke at this and see what I can come up with.

Thanks for the info.

elsigh · 2015-10-15T16:38:17Z

Oh, if I should not have merged this it's my fault. @mattrobenolt Can you make a rollback PR or otherwise address @crbunney's comments? I'm really just an admin and not really opinionated about the development of uap-python at this point. If either of you wants to get more involved and review PR's, wrangle about overall design etc.. please let me know!

mattrobenolt · 2015-10-15T16:42:04Z

@elsigh Comments addressed.

As I mentioned, this does directly conflict, but IMO, #12 is more of a breaking change since it breaks any possibility of pushing a new release to pypi.

I think there's a way we can solve both, which is addressed in #27 (comment)

But I haven't explored since I didn't think of that use case when I put up this PR, nor did I see his attempt to fix. I just noticed that it was broken on HEAD.

crbunney · 2015-10-15T16:53:16Z

but IMO, #12 is more of a breaking change since it breaks any possibility of pushing a new release to pypi.

nor did I see his attempt to fix. I just noticed that it was broken on HEAD.

Embarrassingly, I didn't realise there was a problem with it until I read your comment above. EDIT: Turns out I didn't read the original description closely enough, because you explained it plainly there as well

Oh, if I should not have merged this it's my fault.

In truth, now that @mattrobenolt has pointed it out, it's my pr (#12) that is the problem not this one. #27 has simply fixed what I inadvertently broke

mattrobenolt reviewed Oct 11, 2015
View reviewed changes

mattrobenolt mentioned this pull request Oct 11, 2015

Pre-compile yaml into py file #23

Closed

mattrobenolt force-pushed the install branch 3 times, most recently from 40cebf2 to de60ab9 Compare October 11, 2015 11:15

Fix install from sdist and bdist_wheel

7f2e72f

mattrobenolt force-pushed the install branch from de60ab9 to 7f2e72f Compare October 11, 2015 11:22

Don't allow an IOError, bubble it up since it's required to exist

297b039

selwin pushed a commit to selwin/uap-python that referenced this pull request Oct 14, 2015

Merge pull request ua-parser#27 from commenthol/better-bot-detection

7977294

better bot detection

elsigh added a commit that referenced this pull request Oct 15, 2015

Merge pull request #27 from mattrobenolt/install

8bff839

Fix install from sdist and bdist_wheel

elsigh merged commit 8bff839 into ua-parser:master Oct 15, 2015

mattrobenolt deleted the install branch October 15, 2015 16:32

crbunney reviewed Oct 15, 2015
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix install from sdist and bdist_wheel #27

Fix install from sdist and bdist_wheel #27

Uh oh!

mattrobenolt commented Oct 11, 2015

Uh oh!

mattrobenolt Oct 11, 2015

Uh oh!

mattrobenolt commented Oct 11, 2015

Uh oh!

mattrobenolt commented Oct 11, 2015

Uh oh!

elsigh commented Oct 15, 2015

Uh oh!

crbunney Oct 15, 2015

Uh oh!

mattrobenolt Oct 15, 2015

Uh oh!

mattrobenolt Oct 15, 2015

Uh oh!

crbunney Oct 15, 2015

Uh oh!

mattrobenolt Oct 15, 2015

Uh oh!

crbunney Oct 15, 2015

Uh oh!

mattrobenolt Oct 15, 2015

Uh oh!

elsigh commented Oct 15, 2015

Uh oh!

mattrobenolt commented Oct 15, 2015

Uh oh!

crbunney commented Oct 15, 2015

Uh oh!

Uh oh!

		@@ -38,11 +35,6 @@ def run(self):
		_sdist.run(self)


		class install(_install):

Fix install from sdist and bdist_wheel #27

Fix install from sdist and bdist_wheel #27

Uh oh!

Conversation

mattrobenolt commented Oct 11, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattrobenolt commented Oct 11, 2015

Uh oh!

mattrobenolt commented Oct 11, 2015

Uh oh!

elsigh commented Oct 15, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elsigh commented Oct 15, 2015

Uh oh!

mattrobenolt commented Oct 15, 2015

Uh oh!

crbunney commented Oct 15, 2015

Uh oh!

Uh oh!