Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Back-reference optimizations #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 2, 2017
Merged

Conversation

QuLogic
Copy link
Contributor

@QuLogic QuLogic commented Jul 29, 2017

Finding back-references involves downloading a file for every possibly used object in order to do two things:

  1. Check for the correct extension
  2. Check for the correct anchor

All this downloading makes back-referencing slow, so instead:

  1. Download the index page, parse out the DOCUMENTATION_OPTIONS and find the FILE_SUFFIX from there. If it's not defined, it's a really old Sphinx so assume .rst.html.
  2. Use the same method of finding the anchor as the JavaScript search code.

With these two changes, we can stop downloading all those files and just download 2 per back-reference URL. Additionally, use Sphinx's built-in parser to cut down on the parsing code.

if full_name in self._searchindex['objects']:
value = self._searchindex['objects'][full_name]
if isinstance(value, dict):
value = value[next(iter(value.keys()))]
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is completely wrong. For example, in plot_gallery_version, np.random.random is linked to np.random.lognormal. I believe this is also in part due to #273.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tested this PR. but in this case it does not even recognize np.random.random I'll see with #273

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely should be because of #273.

Copy link
Member

@Titan-C Titan-C left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. Thank you for doing this. I would love to have a test for the _get_link, not sure how to do it yet.

else:
value = int(value)

docopts[key] = value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make a test for this function. Or at least move the parse to a separate function and then test that part at least.

Copy link
Contributor Author

@QuLogic QuLogic Jul 30, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test added, but this parsing is a separate function already?

@@ -30,6 +30,9 @@

from io import StringIO

from sphinx.search import js_index
from sphinx.util import jsdump
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not used

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed.

if full_name in self._searchindex['objects']:
value = self._searchindex['objects'][full_name]
if isinstance(value, dict):
value = value[next(iter(value.keys()))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tested this PR. but in this case it does not even recognize np.random.random I'll see with #273

else:
value = int(value)

docopts[key] = value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you provide a test for this function. Or at least move this parse to an independent function an test that part at least.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for tests

@choldgraf
Copy link
Contributor

this sounds like a great addition, thanks @QuLogic ... I don't have a ton of experience w/ intersphinx code but I agree that it'd be good to test any new parsing functionality built in here.

@choldgraf
Copy link
Contributor

just a ping that #273 is merged now!

@Titan-C
Copy link
Member

Titan-C commented Aug 1, 2017

This works well on my machine. I relaunched Circle CI, but it did not make the build on top of master to include #273, so I could not check on the other versions.
+1 after a rebase, just to check.

@larsoner
Copy link
Contributor

larsoner commented Aug 1, 2017

On my OSX machine, on current master it gets stuck on the embedding step. With this PR things at least don't hang, so +1 for merge from my end!

QuLogic added 6 commits August 1, 2017 17:18
This removes a lot of fragile manual parsing that is already maintained
elsewhere.
There are only two callers and neither use these options.
This uses the data in the search index exactly as the JavaScript code
does, meaning there's no need to search through the HTML of the page to
check for the anchor.
Otherwise, it's very vague what URL is failing.
@QuLogic
Copy link
Contributor Author

QuLogic commented Aug 1, 2017

Rebased and np.random.random is now pointing to the correct function here.

@Titan-C Titan-C merged commit 525120e into sphinx-gallery:master Aug 2, 2017
@Titan-C
Copy link
Member

Titan-C commented Aug 2, 2017

Merged. Thanks @QuLogic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants