ENH: (free-threading Python) avoid C APIs returning borrowed references (⏰ wait for discussion #16839) #16834
Conversation
|
Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.
|
|
👋 Thank you for your draft pull request! Do you know that you can use |
There was a problem hiding this comment.
I like getting us towards free-threading, but this seems a little too ad-hoc. My sense is that we should look at what numpy is doing -- numpy/numpy#26157 -- and see if we can follow their approach. Note that they vendor pythoncapi-compat so that they can just change code, rather than have python version dependent stuff all over. I think we should follow a similar approach (since we built against numpy, perhaps we do not even have to vendor it ourselves?).
This is actually how I got here: I was following along numpy/numpy#27145 😅 |
I cannot help but feel this could be overkill at this point, but maybe worth opening an issue about it now ? |
|
Maybe we can also |
376febd to
18779ca
Compare
|
I've updated the patch with the missing |
There was a problem hiding this comment.
This isn't quite right still for the PyDict_GetItemStringRef and also the actual code is not quite right anyway, since the imported module should be DECREF'd. Not that it would matter much, since if astropy.units disappears, there are larger problems than thread safety...
Somewhat similarly, for the two PyList_GetItemRef, I think the fix is not particularly useful, since if there were multiple threads mangling the list, the code would still fail anyway.
Overall, remain unsure this is worth doing.
astropy/wcs/src/unit_list_proxy.c
Outdated
| @@ -111,6 +114,10 @@ PyUnitListProxy_New( | |||
| self->size = size; | |||
| self->array = array; | |||
| self->unit_class = unit_class; | |||
|
|
|||
| #if PY_VERSION_HEX >= 0x030d00c1 | |||
| Py_DECREF(unit_class); | |||
There was a problem hiding this comment.
Instead, do not Py_INCREF(unit_class) on l. 104
astropy/wcs/src/unit_list_proxy.c
Outdated
| @@ -91,8 +91,11 @@ PyUnitListProxy_New( | |||
| if (units_dict == NULL) { | |||
| return NULL; | |||
| } | |||
|
|
|||
| #if PY_VERSION_HEX >= 0x030d00c1 | |||
| PyDict_GetItemStringRef(units_dict, "Unit", &unit_class); | |||
There was a problem hiding this comment.
This still is not quite right: PyDict_GetItemStringRef returns an int that is 1 if the entry is found, 0 if not, -1 on error. One should check this. Indeed, on error, unit_class is not set, so the current check would fail (since unit_class is not initialized to NULL).
| @@ -341,7 +341,13 @@ static int PyCelprm_set_ref(PyCelprm* self, PyObject* value, void* closure) | |||
|
|
|||
| if (PyList_Check(value)) { | |||
| for (i = 0; i < size; i++) { | |||
| #if PY_VERSION_HEX >= 0x030d00c1 | |||
There was a problem hiding this comment.
Here, it is a bit more of a logic question: this list already has been turned into an array. If there is any risk of another thread mangling it during our little loop here, surely the risk that it is mangled between being turned into an array and this piece of code is even larger. So, really, the list would need to be locked, or, perhaps better, the conversion to NPY_DOUBLE should be done in a second stage, after plain conversion to an array (which would be NPY_OBJECT if None were present.
Separately, if we are really worried about other threads mangling the list, we should check for errors...
There was a problem hiding this comment.
I think I need to learn about how locking single objects works in the C API before I can properly respond here. I'll add it to my list !
| @@ -571,7 +571,13 @@ static int PyPrjprm_set_pv(PyPrjprm* self, PyObject* value, void* closure) | |||
|
|
|||
| if (PyList_Check(value)) { | |||
| for (k = 0; k < size; k++) { | |||
| #if PY_VERSION_HEX >= 0x030d00c1 | |||
There was a problem hiding this comment.
Same as above, though seeing this done twice, it also would seem better to factor this out and make a separate function that returns a MaskedArray or so. To be honest, I think this is best done outside of c.
|
This is not nearly converged enough to make the cut. In fact I'm not even sure we should aim to merge this for 7.1 so let's just remove the milestone for now. |
|
Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then. In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary. If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock. If you believe I commented on this pull request incorrectly, please report this here. |
|
I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks! If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here. |
Description
Although a bit early to start testing on the free-threaded build of Python 3.13, my prep involved reading through https://docs.python.org/3.13/howto/free-threading-extensions.html#borrowed-references, so I gave it a look and found only 3 lines of code using APIs that return borrowed reference (thread unsafe), so it seemed worth fixing them while I was at it.
Blocked by
pythoncapi-compat? #16839