-
-
Notifications
You must be signed in to change notification settings - Fork 11.7k
MAINT: Take advantage of C++ for more robust use of the PyGILState_* API. #30180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
If we decide to add this (or something like it), we'll also need to decide if we will make it part of |
mattip
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice. Do we actually have any code that will throw a C++ exception? If so that should definitely use this construct and not the NPY_ALLOW_C_API_DEF macros
|
I don't think we have a C++ API? We have a C API that can be used from C++, I don't think we have any public API elements that are C++ only. The insertion of this Creating a new header-only C++ API with functionality that's C++-only seems like a large decision. I'd lean to no, but if we'd go that way than it would at a minimum need a separate header file. |
🤦 Yeah, I'll move it. To address the comments about "public C++ API", I'll put it someplace private. Whether or not to establish a public C++ API, and where to put it if we do (and how to name things, including a possible namespace) can be decided later. |
|
Yeah, I agree that this should just not be public, that GIL stuff feels like not our job and easily stolen. I would not mind C++ header only API, but I don't really feel like doing it for Python things. I am also a bit confused, while I think this makes sense, I think we need a RAII pattern for a EDIT: OK, I see the current code seems a bit messy/unclear, which is why ensure GIL is the more obvious stary. |
| EnsureGIL ensure_gil{}; | ||
| descr = PyArray_DESCR(self); | ||
| Py_INCREF(descr); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You probably already on it, but the thing that rubs me wrong about this, is that all of these places shouldn't exist to begin with, I think.
Replacing the finally action with a full-blown RAII pattern will be an improvement! But the thing is that for example this block is useless and because of this obfuscating what is simple into something that seems difficult, IMO.
unique_* are called with the GIL held and defensively ensuring we hold it just obfuscates the code, one could assert that if we must, but that would be the most
.
Now, there is the annoyance that we release the GIL twice within the function, but that is still far better done via:
// code with gil
{
no_gil release via RAII
// code without GIL
}
// code with gil
{
no_gil release via RAII
}
// code with gil
Somewhere along the line, what was simple enough restoring either got messed up and then apparently these defensive GIL ensuring things were added.
As a rule of thumb though, defensively grabbing the GIL only makes sense for setting an error, and we even have npy_gil_error() for that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my next update, I will have a (very conventional) RAII class that wraps PyEval_SaveThread() and PyEval_RestoreThread(thread_state) using the same constructor/destructor pattern as EnsureGIL, which I think is what you are suggesting above.
I agree that if we are sure the various unique functions are always called with the GIL (which is reasonable--unique is not a ufunc, after all), then the current use of NPY_ALLOW_C_API, etc, in there is unnecessary. I'll remove that in the next update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My updates in unique.cpp are becoming nontrivial: in addition to removing the use of the macros related to ensuring the GIL (NPY_ALLOW_C_API, etc) and adding the use of RAII objects for PyEval_SaveThread/_RestoreThread management, there is some incidental DRY clean up and encapsulation that works out nicely. That's not a problem, but these changes will cause big conflicts with @math-hiyoko's recent PR to use a hash map to implement the various return_ options of unique (#30110). I haven't looked at that PR closely yet, but if it looks good enough to eventually merge, we should get that in before I pursue my PR here. I suspect that at least some of what I am thinking of changing could be incorporated into #30110.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, thanks @math-hiyoko. I'll continue working on this, and keep an eye on #30110.
In C++ code, instead of writing
we can write
This ensures that
PyGILState_Release(__save__)is called, even if the wrapped code throws an exception or executes a return or a goto. The latter currently occurs in a few places in our code.