ENH: improve Timsort with powersort merge-policy #29208
Conversation
Implement the improved merge policy for Timsort, as developed by Munro and Wild. Benchmarks show a significant improvement in performance.
|
The test failures all look lint related. |
|
Hi, I might need some help understanding what the failing checks are about. Linux tests / lint (pull_request) shows 2 errors concerning thanks for the quick response! |
|
Probably #29210. If so, a rebase should fix it. |
|
We aren't able to run ruff on the differences yet, but it is fast enough that we run it on the whole project. Not sure how that failure slipped through. |
It must've been an outdated branch that was merged |
|
Thanks @moritz-gross . Note that this is safe as long as the names/api in the sort library are not changed. The indirect sort versions would be good to have. In general, all types are desirable, but note that the short integer types are actually handled by radix sort. Is the much to be gained for those types? Argsorts have the disadvantage of accessing memory in a scattered matter and might benefit from handling long runs. I suppose the real tradeoff is clarity in what algorithms are used and where. We should probably put together a table of which algorithms handle which types, and another for which algorithms are actually used by NumPy for the different types and sort kinds. |
our contribution
This PR implements Powersort, a sorting algorithm based on the currently used Timsort that has already been implemented in CPython and PyPy.
We benchmarked the implementation on collected data and archieved a strong improvement (in particular on large instances with length over 1 million):
Powersort is a drop-in replacement with the same functionality as Timsort, which we tested with
spin test -m full(though onlyspin test -- -k "test_multiarray"should be relevant).open questions