Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Conversation

@WarrenWeckesser
Copy link
Member

In C++ code, instead of writing

NPY_ALLOW_C_API_DEF
NPY_ALLOW_C_API
[code that uses the Python C API here]
NPY_DISABLE_C_API

we can write

{
    EnsureGIL ensure_gil{};
    [code that uses the Python C API here]
}

This ensures that PyGILState_Release(__save__) is called, even if the wrapped code throws an exception or executes a return or a goto. The latter currently occurs in a few places in our code.

@WarrenWeckesser WarrenWeckesser added the C++ Related to introduction and use of C++ in the NumPy code base label Nov 8, 2025
@WarrenWeckesser
Copy link
Member Author

WarrenWeckesser commented Nov 8, 2025

If we decide to add this (or something like it), we'll also need to decide if we will make it part of the a public C++ API, and document like we do the C macros.

Copy link
Member

@mattip mattip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Do we actually have any code that will throw a C++ exception? If so that should definitely use this construct and not the NPY_ALLOW_C_API_DEF macros

@rgommers
Copy link
Member

rgommers commented Nov 9, 2025

I don't think we have a C++ API? We have a C API that can be used from C++, I don't think we have any public API elements that are C++ only.

The insertion of this EnsureGIL code is in the wrong place even if we do decide to make it public - you inserted it in the middle of the code defining npy_cdouble & co which happened to need an #if defined(__cplusplus) for struct types.

Creating a new header-only C++ API with functionality that's C++-only seems like a large decision. I'd lean to no, but if we'd go that way than it would at a minimum need a separate header file.

@WarrenWeckesser
Copy link
Member Author

The insertion of this EnsureGIL code is in the wrong place ...

🤦 Yeah, I'll move it. To address the comments about "public C++ API", I'll put it someplace private. Whether or not to establish a public C++ API, and where to put it if we do (and how to name things, including a possible namespace) can be decided later.

@seberg
Copy link
Member

seberg commented Nov 9, 2025

Yeah, I agree that this should just not be public, that GIL stuff feels like not our job and easily stolen. I would not mind C++ header only API, but I don't really feel like doing it for Python things.
(It may make sense to start such a thing outside NumPy. I have thought that it could be nice to allow re-use of NumPy patterns e.g. to define sorts/ufuncs for user DTypes.)

I am also a bit confused, while I think this makes sense, I think we need a RAII pattern for a nogil block rather than one for ensure GIL (which is usually more confined to error handling anyway and we have a helper for that)?

EDIT: OK, I see the current code seems a bit messy/unclear, which is why ensure GIL is the more obvious stary.

EnsureGIL ensure_gil{};
descr = PyArray_DESCR(self);
Py_INCREF(descr);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably already on it, but the thing that rubs me wrong about this, is that all of these places shouldn't exist to begin with, I think.

Replacing the finally action with a full-blown RAII pattern will be an improvement! But the thing is that for example this block is useless and because of this obfuscating what is simple into something that seems difficult, IMO.

unique_* are called with the GIL held and defensively ensuring we hold it just obfuscates the code, one could assert that if we must, but that would be the most
.
Now, there is the annoyance that we release the GIL twice within the function, but that is still far better done via:

// code with gil
{
    no_gil release via RAII
    // code without GIL
}
// code with gil
{
    no_gil release via RAII
}
// code with gil

Somewhere along the line, what was simple enough restoring either got messed up and then apparently these defensive GIL ensuring things were added.
As a rule of thumb though, defensively grabbing the GIL only makes sense for setting an error, and we even have npy_gil_error() for that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my next update, I will have a (very conventional) RAII class that wraps PyEval_SaveThread() and PyEval_RestoreThread(thread_state) using the same constructor/destructor pattern as EnsureGIL, which I think is what you are suggesting above.

I agree that if we are sure the various unique functions are always called with the GIL (which is reasonable--unique is not a ufunc, after all), then the current use of NPY_ALLOW_C_API, etc, in there is unnecessary. I'll remove that in the next update.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My updates in unique.cpp are becoming nontrivial: in addition to removing the use of the macros related to ensuring the GIL (NPY_ALLOW_C_API, etc) and adding the use of RAII objects for PyEval_SaveThread/_RestoreThread management, there is some incidental DRY clean up and encapsulation that works out nicely. That's not a problem, but these changes will cause big conflicts with @math-hiyoko's recent PR to use a hash map to implement the various return_ options of unique (#30110). I haven't looked at that PR closely yet, but if it looks good enough to eventually merge, we should get that in before I pursue my PR here. I suspect that at least some of what I am thinking of changing could be incorporated into #30110.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the consideration.
#30110 is still under active development and there are several parts that I plan to revise, so waiting for it to be completed might delay this PR.

I would prefer to have this PR merged first, and then update #30110 to incorporate the changes from it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks @math-hiyoko. I'll continue working on this, and keep an eye on #30110.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

03 - Maintenance C++ Related to introduction and use of C++ in the NumPy code base

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants