Add Cython implementation for VOTable binary converters #18454
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
82cd430 to
1dfea18
Compare
pllim
reviewed
Jul 23, 2025
| @@ -842,6 +828,12 @@ class Double(FloatingPoint): | |||
|
|
|||
| format = "f8" | |||
|
|
|||
| def binparse(self, read): | |||
There was a problem hiding this comment.
Seems a little repetitive across the board. Looks like the only difference is a number. Can this method be inherited but access some self._expected_len (name negotiable) that is set by the subclass?
There was a problem hiding this comment.
Great suggestion, looking back at it now I can see there is a lot of duplication. I've moved things up to the parent Numeric class, let me know if that looks good. (Or if you see any adjustments needing to be made)
pllim
reviewed
Jul 23, 2025
There was a problem hiding this comment.
Once we have the benchmark down, should maybe put some ballpark values here.
1 task
|
Hmm looks like @eerovaher proposed an alternative at #18455 . Are you interested to have a look, @stvoutsin ? Thanks, all! |
|
astropy/astropy-benchmarks#141 is merged so theoretically next push would trigger a new benchmark run with the new relevant benchmarks. 🤞 |
f6f4267 to
11d9403
Compare
ba78237 to
1babe2e
Compare
|
Hmm benchmark job says nothing has "significantly changed". Is this expected? I thought it would say "this and that is much faster" or something. |
Apologies I think i introduced a regression while trying to fix an issue that caused failed tests in the CI. I will also test with increasing the row sizes for the sample data to see how the runtime difference between main and this branch scales. |
|
I've run the astropy-benchmarks locally with 200k, 500k and 1M sample row sizes (https://github.com/astropy/astropy-benchmarks/blob/main/benchmarks/votable.py#L13). Results shown here: u/stvoutsin/binary2-cython vs main branch performance comparison
|
350e752 to
96b3904
Compare
|
Benchmark run looking good. Thanks!
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request attempts to improve the performance of VOTable binary parsing by implementing optimized Cython converters.
Initial benchmarks have shown performance improvements of the order:
Problem:
We’re seeing relatively slow performance when parsing large VOTables using the BINARY2 serialization format.
Profiling results from py-spy show a significant portion of time is spent in astropy.io.votable.converters.binparse. Here’s an excerpt of the profiler output:
As shown, the hot path is dominated by the binparse function in converters.py.
Solution:
double,float,int,long,short,unsignedByte,boolean,bit)Compatibility:
Should be 100% backward compatible since the API has not been modified
Testing:
Added comprehensive benchmarks to astropy-benchmarks showing consistent 30-50% performance improvements across different table types and data patterns. astropy/astropy-benchmarks#141
Astropy:main
u/stvoutsin/binary2-cython
For better visualization:
I've also tested parsing outside of the astropy-benchmark PR with a VOTable with 50000 rows and aprox. 1000 columns (~50 million cells).
With this table the parsing took 30 seconds, down from 65 seconds in the version currently on main.
Fixes #18442
Any thoughts on this approach? Are there alternatives that I haven't considered? Perhaps someone with more Cython experience can let me know if I've made any obvious mistakes or if there are better way of doing any of this.