Move exc state to generator. Fixes bpo-25612 #1773

markshannon · 2017-05-23T22:08:53Z

Moves the exception state from the frame to the coroutine. (Coroutine in the CS sense, which includes generators, coroutines and thread). This cleans up the code, removing about 60 lines, and fixes https://bugs.python.org/issue25612.

https://bugs.python.org/issue25612

the-knights-who-say-ni · 2017-05-23T22:08:55Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

mention-bot · 2017-05-23T22:08:59Z

@markshannon, thanks for your PR! By analyzing the history of the files in this pull request, we identified @benjaminp, @serhiy-storchaka and @Yhg1s to be potential reviewers.

markshannon · 2017-05-23T22:13:05Z

I've just linked by bpo user to my github user. Which should satisfy @the-knights-who-say-ni, hopefully.

njsmith · 2017-05-23T23:02:22Z

You might also want to check how this affects this strange behavior that @arigo found: https://bugs.python.org/issue28884#msg282532

1st1

I like this refactoring a lot. I'd go a step further and refactor tstate->*curexc_ too as well as add a couple of helper macros (not defined in limited API).

1st1 · 2017-05-23T23:23:01Z

Objects/genobject.c

@@ -115,6 +118,9 @@ gen_dealloc(PyGenObject *gen)
    Py_CLEAR(gen->gi_code);
    Py_CLEAR(gen->gi_name);
    Py_CLEAR(gen->gi_qualname);
+    Py_CLEAR(gen->gi_exc_state.exc_type);


Maybe we should add a macro PyExcState_CLEAR (and _VISIT)?

Done (although I used inline functions)

1st1 · 2017-05-23T23:26:39Z

Include/pystate.h

-    PyObject *exc_type;
-    PyObject *exc_value;
-    PyObject *exc_traceback;
+    PyExcState exc_state;


Given that this is already quite an invasive change, maybe replace curexc_* fields with curexc_state?

PyExcState structs form a stack (hence the "previous" field), but there is only one current exception.

1st1 · 2017-05-23T23:28:11Z

Include/pystate.h

@@ -150,6 +162,8 @@ typedef struct _ts {
    PyObject *async_gen_firstiter;
    PyObject *async_gen_finalizer;

+    PyExcState *exc_info;


Can we have exc_info, exc_state and curexc_* defined next to each other with comments explaining how we use each one of them? Since you've already spent time to figure this stuff out anyways.

1st1 · 2017-05-23T23:29:39Z

Objects/genobject.c

-        f->f_exc_type = NULL;
-        f->f_exc_value = NULL;
-        f->f_exc_traceback = NULL;
+        t = gen->gi_exc_state.exc_type;


If we had a PyExcState_CLEAR macro we could use it here.

markshannon · 2017-05-23T23:35:58Z

@njsmith It does indeed fix https://bugs.python.org/issue28884#msg282532. I've added it as a test case.

1st1 · 2017-05-25T04:40:54Z

Include/pystate.h

-    PyObject *exc_type;
-    PyObject *exc_value;
-    PyObject *exc_traceback;
+    /* The exception currently being handled, if no coroutines/generators are present.


There are many lines that are longer than 79 (see PEP 7). Could you please fix this? </nitpicking>

1st1 · 2017-05-25T04:41:50Z

Python/errors.c

+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate) {
+    PyExcState *exc_info = tstate->exc_info;
+    while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
+           exc_info->exc_previous != NULL)


Another nit, per PEP 7 we now want to use { even for blocks with one statement.

1st1 · 2017-05-25T04:43:46Z

Aside from a few code formatting nits the patch looks good. I just want to test it on my machine before giving it a green light; will do that tomorrow or the day after.

1st1 · 2017-05-26T20:22:34Z

Objects/genobject.c

@@ -86,6 +93,19 @@ _PyGen_Finalize(PyObject *self)
    PyErr_Restore(error_type, error_value, error_traceback);
 }

+static inline void PyExcState_clear(PyExcState *exc_state) {


Another minor nit: please rename to PyExcState_Clear.

larryhastings · 2017-07-15T15:22:36Z

Include/pystate.h

+       state inside the generator and the exception state of the calling
+       frame (which shouldn't be impacted when the generator "yields"
+       from an except handler).
+       These three fields exist exactly for that. */


Should this comment be left verbatim like this? The comment applied when these fields were part of the frame object, but they've been removed from the frame object completely. I don't claim to understand the patch completely but I assume the use of these fields have changed somewhat--otherwise what's the point of shuffling the data around?

larryhastings · 2017-07-15T15:31:29Z

Include/pystate.h

+
+    struct _excstate *exc_previous;
+
+} PyExcState;


Would you consider a longer name here? I always remember MvL saying "avoid abbreviations in identifiers, they are mystifying to non-English speakers". "PyExceptionState" isn't that awful is it?

If this name is not public, it should be underscored.

A short name would LGTM, but the PyExc prefix is used only for exception classes. The PyException prefix is used for manipulating individual exception objects (PyExceptionClass_Check, PyException_GetTraceback). All other names related to exceptions and error handling have prefix PyErr.

PyExcState sounds ok to me.

1st1 · 2017-10-10T23:18:11Z

@pitrou Antoine, could you please take a look? I'm inclined to merge this.

@markshannon Could you please rebase and add a NEWS entry? Would be cool if you could address the few review nits, but if you don't have time I can merge this as is and fix them myself.

serhiy-storchaka · 2017-10-11T06:31:26Z

Include/pyerrors.h

@@ -78,6 +78,7 @@ PyAPI_FUNC(void) PyErr_SetNone(PyObject *);
 PyAPI_FUNC(void) PyErr_SetObject(PyObject *, PyObject *);
 #ifndef Py_LIMITED_API
 PyAPI_FUNC(void) _PyErr_SetKeyError(PyObject *);
+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate);


PyAPI_FUNC()

This isn't intended to be an API function. It is only used in errors.c and sysmodule.c

Even for private functions we still use PyAPI_FUNC by convention (such as _PyErr_SetKeyError just above).

But it isn't meant to be part of an API. What does PyAPI_FUNC mean?

Even for private functions we still use PyAPI_FUNC by convention (such as _PyErr_SetKeyError just above).

See _PyCoro_GetAwaitableIter and _PyGen_yf etc. I agree with @markshannon here, we don't really need PyAPI_FUNC.

serhiy-storchaka · 2017-10-11T06:33:58Z

Include/pystate.h

+
+    struct _excstate *exc_previous;
+
+} PyExcState;


If this name is not public, it should be underscored.

A short name would LGTM, but the PyExc prefix is used only for exception classes. The PyException prefix is used for manipulating individual exception objects (PyExceptionClass_Check, PyException_GetTraceback). All other names related to exceptions and error handling have prefix PyErr.

serhiy-storchaka · 2017-10-11T06:37:23Z

Objects/genobject.c

@@ -15,14 +15,22 @@ static char *NON_INIT_CORO_MSG = "can't send non-None value to a "
 static char *ASYNC_GEN_IGNORED_EXIT_MSG =
                                 "async generator ignored GeneratorExit";

+static inline int


Py_LOCAL_INLINE(int)?

Now that we support C99, I think we ought to use standard C where possible.

serhiy-storchaka · 2017-10-11T06:37:47Z

Objects/genobject.c

@@ -15,14 +15,22 @@ static char *NON_INIT_CORO_MSG = "can't send non-None value to a "
 static char *ASYNC_GEN_IGNORED_EXIT_MSG =
                                 "async generator ignored GeneratorExit";

+static inline int
+PyExcState_traverse(PyExcState *exc_state, visitproc visit, void *arg) {


Move { on a separate line.

If PyExcState_traverse is private and local use the all-lowercase-letters style.

serhiy-storchaka · 2017-10-11T06:39:55Z

Objects/genobject.c

@@ -86,6 +94,19 @@ _PyGen_Finalize(PyObject *self)
    PyErr_Restore(error_type, error_value, error_traceback);
 }

+static inline void PyExcState_Clear(PyExcState *exc_state) {


Move static inline void (or Py_LOCAL_INLINE(int)?) and { on separate lines.

If PyExcState_Clear is private and local use the all-lowercase-letters style.

serhiy-storchaka · 2017-10-11T07:11:38Z

Python/errors.c

@@ -52,6 +52,15 @@ PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
    Py_XDECREF(oldtraceback);
 }

+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate) {


Move PyExcState * and { on separate lines.

serhiy-storchaka · 2017-10-11T07:25:10Z

Python/errors.c

+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate) {
+    PyExcState *exc_info = tstate->exc_info;
+    while ((exc_info->exc_type == NULL || exc_info->exc_type == Py_None) &&
+           exc_info->exc_previous != NULL) {


Move { on a separate line (new PEP 7 rule for multiline conditions).

serhiy-storchaka · 2017-10-11T07:36:41Z

Python/pystate.c

+    Py_CLEAR(tstate->exc_state.exc_value);
+    Py_CLEAR(tstate->exc_state.exc_traceback);
+    assert(tstate->exc_info->exc_previous == NULL);
+    if (Py_VerboseFlag && tstate->exc_info != &tstate->exc_state) {


Py_VerboseFlag only is used for import related tracing.

This is consistent with https://github.com/python/cpython/blob/master/Python/pystate.c#L434. I'm not sure what is the ideal behaviour, but it ought to be consistent.

Then that code should be fixed too.

serhiy-storchaka · 2017-10-11T07:36:59Z

Python/pystate.c

+    Py_CLEAR(tstate->exc_state.exc_traceback);
+    assert(tstate->exc_info->exc_previous == NULL);
+    if (Py_VerboseFlag && tstate->exc_info != &tstate->exc_state) {
+        fprintf(stderr,


Can PyErr_WarnEx() or PySys_WriteStderr() be used here?

serhiy-storchaka · 2017-10-11T08:00:03Z

Python/errors.c

@@ -52,6 +52,15 @@ PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
    Py_XDECREF(oldtraceback);
 }

+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate) {


The result of this function is read-only, and exc_previous is not used. Wouldn't be better to return just three objects like in PyErr_GetExcInfo()? This would make the PyExcState structure more private.

If I made it return three objects, it would be the same as PyErr_GetExcInfo().
_PyErr_GetExcInfo() is provided as a way to handle the type, value, traceback trio in a reasonably modular fashion, without to much overhead.
As it is only used in errors.c and sysmodule.c, it is effectively private.

If it would be the same as PyErr_GetExcInfo(), we should just use PyErr_GetExcInfo().

pitrou · 2017-10-11T10:36:05Z

Python/pystate.c

+    Py_CLEAR(tstate->exc_state.exc_type);
+    Py_CLEAR(tstate->exc_state.exc_value);
+    Py_CLEAR(tstate->exc_state.exc_traceback);
+    assert(tstate->exc_info->exc_previous == NULL);


Add a comment here?

pitrou · 2017-10-11T10:37:21Z

This looks good to me on the principle. It needs adressing @serhiy-storchaka 's comments, and also fixing the merge conflicts.

1st1 · 2017-10-11T20:45:15Z

@markshannon Mark, will you be able to work on this in the following couple of weeks? If not, I can help resolve code review comments.

markshannon · 2017-10-12T09:13:51Z

Not right now, but in the next week or so, yes.

…erator/thread) object where it belongs.

…t exception 'state' forms a stack.

pitrou · 2017-10-22T14:00:29Z

@markshannon, do you have push rights or would you like someone else to merge this PR?

serhiy-storchaka · 2017-10-22T14:05:16Z

Include/pystate.h

+
+    struct _exc_stackitem *exc_previous;
+
+} _PyErr_StackItem;


Inconsistent names _exc_stackitem and _PyErr_StackItem.

serhiy-storchaka · 2017-10-22T14:07:53Z

Objects/genobject.c

@@ -87,6 +96,20 @@ _PyGen_Finalize(PyObject *self)
    PyErr_Restore(error_type, error_value, error_traceback);
 }

+static inline void exc_state_clear(_PyErr_StackItem *exc_state)


Move static inline void on a separate line.

serhiy-storchaka · 2017-10-22T14:10:28Z

Python/errors.c

@@ -53,6 +53,17 @@ PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
    Py_XDECREF(oldtraceback);
 }

+_PyErr_StackItem *_PyErr_GetTopmostException(PyThreadState *tstate)


Move _PyErr_StackItem * on a separate line.

serhiy-storchaka · 2017-10-22T14:12:41Z

Python/errors.c

@@ -52,6 +52,15 @@ PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
    Py_XDECREF(oldtraceback);
 }

+PyExcState *_PyErr_GetExcInfo(PyThreadState *tstate) {


If it would be the same as PyErr_GetExcInfo(), we should just use PyErr_GetExcInfo().

serhiy-storchaka · 2017-10-22T14:18:26Z

Is the problem I described on the tracker resolved?

markshannon · 2017-10-22T20:26:41Z

@serhiy-storchaka I've just fixed it.
@pitrou I don't have push rights.

1st1 · 2017-10-23T17:57:44Z

Thanks a lot, Mark, for looking into this.

CPython change how the exception information is stored internally in 3.7a3. This simply adds fences based on python version around how to access the objects. Xref: python/cpython@ae3087c https://bugs.python.org/issue25612 python/cpython#1773

bpo-25612 (python#1773) moves the exception state information from frame object to coroutine (generator/thread) object where it belongs. As a consequence Stackless moves the exception state information for non-current tasklets from thread-state to the tasklet. This changes the pickle format of frame, tasklet and generator objects. The commit adds three test cases.

the-knights-who-say-ni added the CLA not signed label May 23, 2017

1st1 self-requested a review May 23, 2017 22:12

1st1 reviewed May 23, 2017

View reviewed changes

markshannon force-pushed the move-exc-state-to-generator branch from e0b55d4 to e19057d Compare May 23, 2017 23:34

the-knights-who-say-ni added CLA signed and removed CLA not signed labels May 24, 2017

1st1 reviewed May 25, 2017

View reviewed changes

1st1 reviewed May 26, 2017

View reviewed changes

markshannon force-pushed the move-exc-state-to-generator branch 2 times, most recently from 0cc764b to 800150c Compare June 11, 2017 11:04

larryhastings reviewed Jul 15, 2017

View reviewed changes

Mariatta added needs rebase and removed needs rebase labels Oct 9, 2017

1st1 requested a review from pitrou October 10, 2017 23:16

1st1 requested a review from serhiy-storchaka October 10, 2017 23:18

serhiy-storchaka reviewed Oct 11, 2017

View reviewed changes

pitrou reviewed Oct 11, 2017

View reviewed changes

Move exception state information from frame objects to coroutine (gen…

adde498

…erator/thread) object where it belongs.

markshannon added 4 commits October 22, 2017 11:31

Add tests to verify fix of bpo-25612 and case B6 of bpo-28884

207ff6c

Factor out clear/traverse for exc_state struct.

52acc3c

Make sure that previous exc-state in thread is initialized properly.

f2ff96e

Make code conform to PEP-7

963aea3

markshannon force-pushed the move-exc-state-to-generator branch from 800150c to 415121e Compare October 22, 2017 12:04

markshannon added 2 commits October 22, 2017 13:18

Change names and formatting to conform to PEP-7 and make explicit tha…

22b1e82

…t exception 'state' forms a stack.

Add blurb entry for new exception state handling.

e5ffc67

markshannon force-pushed the move-exc-state-to-generator branch from 7929841 to e5ffc67 Compare October 22, 2017 12:19

1st1 approved these changes Oct 22, 2017

View reviewed changes

bedevere-bot added the awaiting merge label Oct 22, 2017

Right-justify C macro

d3b7efb

pitrou approved these changes Oct 22, 2017

View reviewed changes

serhiy-storchaka reviewed Oct 22, 2017

View reviewed changes

markshannon added 2 commits October 22, 2017 21:13

More renaming and PEP-7 compliance

36ea52f

Update implementation of 'raise' to scan exception stack.

cba130e

pitrou merged commit ae3087c into python:master Oct 22, 2017

bedevere-bot removed the awaiting merge label Oct 22, 2017

markshannon deleted the move-exc-state-to-generator branch October 23, 2017 07:57

This was referenced Oct 28, 2017

Internals of exceptions on ThreadState _ts struct have moved cython/cython#1955

Closed

FIX: account for change in how exception information is stored cython/cython#1956

Merged

snaury mentioned this pull request Dec 3, 2017

Doesn't compile with Python 3.7 due to tstate changes python-greenlet/greenlet#123

Closed

ghost mentioned this pull request Mar 29, 2019

cothread will not build using Python 3.7 DiamondLightSource/cothread#9

Closed

indrora mentioned this pull request Jun 15, 2019

Fails to build under Python 3.7 seomoz/url-py#63

Closed

Move exc state to generator. Fixes bpo-25612 #1773

Move exc state to generator. Fixes bpo-25612 #1773

Conversation

markshannon commented May 23, 2017 • edited by bedevere-bot Loading

the-knights-who-say-ni commented May 23, 2017

mention-bot commented May 23, 2017

markshannon commented May 23, 2017

njsmith commented May 23, 2017

1st1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markshannon commented May 23, 2017

1st1 May 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1st1 commented May 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1st1 commented Oct 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou commented Oct 11, 2017

1st1 commented Oct 11, 2017

markshannon commented Oct 12, 2017

pitrou commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka commented Oct 22, 2017

markshannon commented Oct 22, 2017

1st1 commented Oct 23, 2017

markshannon commented May 23, 2017 •

edited by bedevere-bot

Loading

1st1 May 25, 2017 •

edited

Loading