Thanks to visit codestin.com
Credit goes to github.com

Skip to content

bpo-45292: [PEP 654] Update traceback display code to work with exception groups #29207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iritkatriel merged 27 commits into from
Nov 5, 2021
Merged

bpo-45292: [PEP 654] Update traceback display code to work with exception groups #29207

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
42bed01
C traceback code
iritkatriel Oct 5, 2021
a2daa23
add ExceptionGroups support to traceback.py
iritkatriel Oct 8, 2021
261917a
remove 'with X sub-exceptions' line from tracebacks
iritkatriel Oct 24, 2021
7613b43
pass margin instead of margin_char
iritkatriel Oct 25, 2021
d98a72b
update news
iritkatriel Oct 25, 2021
d69916e
excs is tuple, use PyTuple apis. Change check to assertion.
iritkatriel Oct 25, 2021
f5cab69
remove redundant num_excs > 0 check (it is asserted above)
iritkatriel Oct 25, 2021
5170f00
remove cpython_only from exception group tests
iritkatriel Oct 25, 2021
2052c77
handle recursion errors (vert deeply nested EGs)
iritkatriel Oct 25, 2021
5097300
WRITE_INDENTED_MARGIN macro --> write_indented_margin function
iritkatriel Oct 26, 2021
dc21cf8
move new traceback utils to internal/
iritkatriel Oct 26, 2021
d4007b7
test improvements
iritkatriel Oct 26, 2021
169934e
pep7, improve error checking and clarity
iritkatriel Oct 26, 2021
aa4da45
add missing test to cover print_chained with/without parent_label
iritkatriel Oct 26, 2021
6ee84f7
compare the complete expected tb text
iritkatriel Oct 26, 2021
ac7f34c
Update Misc/NEWS.d/next/Core and Builtins/2021-09-26-18-18-50.bpo-452…
iritkatriel Oct 26, 2021
d0d4961
don't need the regex anymore
iritkatriel Oct 26, 2021
5c1015d
remove full-path labels
iritkatriel Oct 29, 2021
83abebd
int --> bool
iritkatriel Oct 29, 2021
16d077d
move code around
iritkatriel Oct 29, 2021
64fb164
Tweak the top-level of traceback box as suggested by Yury
iritkatriel Oct 31, 2021
88019f5
tidy up error handling
iritkatriel Nov 1, 2021
e963835
add limits for width and depth of formatted exception groups
iritkatriel Nov 2, 2021
e85510a
use _PyBaseExceptionGroup_Check macro
iritkatriel Nov 2, 2021
c15a7bd
remove redundant PyErr_Clear
iritkatriel Nov 2, 2021
d8cc6e8
minor tweak - move if out of loop
iritkatriel Nov 2, 2021
61fab3f
remove excess whitespace
iritkatriel Nov 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
tidy up error handling
  • Loading branch information
@iritkatriel
iritkatriel committed Nov 1, 2021
commit 88019f57c407923693670bdfae50ee821140e575
139 changes: 85 additions & 54 deletions Python/pythonrun.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -916,12 +916,21 @@ print_exception(struct exception_print_context *ctx, PyObject *value)
_Py_IDENTIFIER(print_file_and_line);

if (!PyExceptionInstance_Check(value)) {
err += _Py_WriteIndent(EXC_INDENT(ctx), f);
err += PyFile_WriteString("TypeError: print_exception(): Exception expected for value, ", f);
err += PyFile_WriteString(Py_TYPE(value)->tp_name, f);
err += PyFile_WriteString(" found\n", f);
if (err)
if (err == 0) {
err = _Py_WriteIndent(EXC_INDENT(ctx), f);
}
if (err == 0) {
err = PyFile_WriteString("TypeError: print_exception(): Exception expected for value, ", f);
}
if (err == 0) {
err = PyFile_WriteString(Py_TYPE(value)->tp_name, f);
}
if (err == 0) {
err = PyFile_WriteString(" found\n", f);
}
if (err != 0) {
PyErr_Clear();
}
return;
}

Expand Down Expand Up @@ -965,9 +974,10 @@ print_exception(struct exception_print_context *ctx, PyObject *value)
filename, lineno);
Py_DECREF(filename);
if (line != NULL) {
err += write_indented_margin(ctx, f);
PyErr_Clear();
PyFile_WriteObject(line, f, Py_PRINT_RAW);
err = write_indented_margin(ctx, f);
if (err == 0) {
err = PyFile_WriteObject(line, f, Py_PRINT_RAW);
}
Py_DECREF(line);
}

Expand Down Expand Up @@ -996,7 +1006,7 @@ print_exception(struct exception_print_context *ctx, PyObject *value)
err = -1;
}
}
if (err) {
if (err != 0) {
/* Don't do anything else */
}
else {
Expand All @@ -1005,23 +1015,26 @@ print_exception(struct exception_print_context *ctx, PyObject *value)
_Py_IDENTIFIER(__module__);
assert(PyExceptionClass_Check(type));

err += write_indented_margin(ctx, f);
modulename = _PyObject_GetAttrId(type, &PyId___module__);
if (modulename == NULL || !PyUnicode_Check(modulename))
{
Py_XDECREF(modulename);
PyErr_Clear();
err += PyFile_WriteString("<unknown>", f);
}
else {
if (!_PyUnicode_EqualToASCIIId(modulename, &PyId_builtins) &&
!_PyUnicode_EqualToASCIIId(modulename, &PyId___main__))
err = write_indented_margin(ctx, f);
if (err == 0) {
modulename = _PyObject_GetAttrId(type, &PyId___module__);
if (modulename == NULL || !PyUnicode_Check(modulename))
{
Py_XDECREF(modulename);
PyErr_Clear();
err += PyFile_WriteObject(modulename, f, Py_PRINT_RAW);
err += PyFile_WriteString(".", f);
err = PyFile_WriteString("<unknown>", f);
}
else {
if (!_PyUnicode_EqualToASCIIId(modulename, &PyId_builtins) &&
!_PyUnicode_EqualToASCIIId(modulename, &PyId___main__))
{
err = PyFile_WriteObject(modulename, f, Py_PRINT_RAW);
if (err == 0) {
err = PyFile_WriteString(".", f);
}
}
Py_DECREF(modulename);
}
Py_DECREF(modulename);
}
if (err == 0) {
PyObject* qualname = PyType_GetQualName((PyTypeObject *)type);
Expand All @@ -1031,7 +1044,6 @@ print_exception(struct exception_print_context *ctx, PyObject *value)
err = PyFile_WriteString("<unknown>", f);
}
else {
PyErr_Clear();
err = PyFile_WriteObject(qualname, f, Py_PRINT_RAW);
Py_DECREF(qualname);
}
Expand Down Expand Up @@ -1098,16 +1110,28 @@ print_chained(struct exception_print_context* ctx, PyObject *value,
bool need_close = ctx->need_close;

int err = Py_EnterRecursiveCall(" in print_chained");
if (!err) {
if (err == 0) {
print_exception_recursive(ctx, value);
Py_LeaveRecursiveCall();

err |= write_indented_margin(ctx, f);
err |= PyFile_WriteString("\n", f);
err |= write_indented_margin(ctx, f);
err |= PyFile_WriteString(message, f);
err |= write_indented_margin(ctx, f);
err |= PyFile_WriteString("\n", f);
if (err == 0) {
err = write_indented_margin(ctx, f);
}
if (err == 0) {
err = PyFile_WriteString("\n", f);
}
if (err == 0) {
err = write_indented_margin(ctx, f);
}
if (err == 0) {
err = PyFile_WriteString(message, f);
}
if (err == 0) {
err = write_indented_margin(ctx, f);
}
if (err == 0) {
err = PyFile_WriteString("\n", f);
}
}
else {
PyErr_Clear();
Expand Down Expand Up @@ -1145,7 +1169,7 @@ print_exception_recursive(struct exception_print_context* ctx, PyObject *value)
if (res == -1)
PyErr_Clear();
if (res == 0) {
err |= print_chained(ctx, cause, cause_message, "cause");
err = print_chained(ctx, cause, cause_message, "cause");
}
}
else if (context &&
Expand All @@ -1160,18 +1184,21 @@ print_exception_recursive(struct exception_print_context* ctx, PyObject *value)
if (res == -1)
PyErr_Clear();
if (res == 0) {
err |= print_chained(ctx, context, context_message, "context");
err = print_chained(ctx, context, context_message, "context");
}
}
Py_XDECREF(context);
Py_XDECREF(cause);
}
Py_XDECREF(value_id);
}
if (!PyObject_TypeCheck(value, (PyTypeObject *)PyExc_BaseExceptionGroup)) {
if (err) {
/* don't do anything else */
}
else if (!PyObject_TypeCheck(value, (PyTypeObject *)PyExc_BaseExceptionGroup)) {
print_exception(ctx, value);
}
else {
else {
/* ExceptionGroup */

/* TODO: add arg to limit number of exceptions printed? */
Expand Down Expand Up @@ -1200,35 +1227,39 @@ print_exception_recursive(struct exception_print_context* ctx, PyObject *value)
(i == 0) ? "+-" : " ", i + 1);

if (line) {
err |= _Py_WriteIndent(EXC_INDENT(ctx), f);
PyErr_Clear();
err |= PyFile_WriteObject(line, f, Py_PRINT_RAW);
err = _Py_WriteIndent(EXC_INDENT(ctx), f);
if (err == 0) {
err = PyFile_WriteObject(line, f, Py_PRINT_RAW);
}
Py_DECREF(line);
}
else {
err = -1;
PyErr_Clear();
}

ctx->exception_group_depth += 1;
PyObject *exc = PyTuple_GET_ITEM(excs, i);
if (err == 0) {
ctx->exception_group_depth += 1;
PyObject *exc = PyTuple_GET_ITEM(excs, i);

if (!Py_EnterRecursiveCall(" in print_exception_recursive")) {
print_exception_recursive(ctx, exc);
Py_LeaveRecursiveCall();
}
else {
err = -1;
PyErr_Clear();
}
if (!Py_EnterRecursiveCall(" in print_exception_recursive")) {
print_exception_recursive(ctx, exc);
Py_LeaveRecursiveCall();
}
else {
err = -1;
}

if (last_exc && ctx->need_close) {
err |= _Py_WriteIndent(EXC_INDENT(ctx), f);
err |= PyFile_WriteString(
"+------------------------------------\n", f);
ctx->need_close = false;
if (err == 0 && last_exc && ctx->need_close) {
err = _Py_WriteIndent(EXC_INDENT(ctx), f);
if (err == 0) {
err = PyFile_WriteString(
"+------------------------------------\n", f);
}
ctx->need_close = false;
}
ctx->exception_group_depth -= 1;
}
ctx->exception_group_depth -= 1;
}
if (ctx->exception_group_depth == 1) {
ctx->exception_group_depth -= 1;
Expand Down
36 changes: 24 additions & 12 deletions Python/traceback.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,8 +408,8 @@ int
_Py_WriteIndentedMargin(int indent, const char *margin, PyObject *f)
{
int err = _Py_WriteIndent(indent, f);
if (margin) {
err |= PyFile_WriteString(margin, f);
if (err == 0 && margin) {
err = PyFile_WriteString(margin, f);
}
return err;
}
Expand Down Expand Up @@ -545,16 +545,22 @@ display_source_line_with_margin(PyObject *f, PyObject *filename, int lineno, int
*truncation = i - indent;
}

err |= _Py_WriteIndentedMargin(margin_indent, margin, f);
if (err == 0) {
err = _Py_WriteIndentedMargin(margin_indent, margin, f);
}
/* Write some spaces before the line */
err |= _Py_WriteIndent(indent, f);
if (err == 0) {
err = _Py_WriteIndent(indent, f);
}

/* finally display the line */
if (err == 0)
if (err == 0) {
err = PyFile_WriteObject(lineobj, f, Py_PRINT_RAW);
}
Py_DECREF(lineobj);
if (err == 0)
if (err == 0) {
err = PyFile_WriteString("\n", f);
}
return err;
}

Expand Down Expand Up @@ -746,7 +752,9 @@ tb_displayline(PyTracebackObject* tb, PyObject *f, PyObject *filename, int linen
if (line == NULL)
return -1;
err = _Py_WriteIndentedMargin(margin_indent, margin, f);
err |= PyFile_WriteObject(line, f, Py_PRINT_RAW);
if (err == 0) {
err = PyFile_WriteObject(line, f, Py_PRINT_RAW);
}
Py_DECREF(line);
if (err != 0)
return err;
Comment on lines +747 to 760
Copy link
Copy Markdown
Member

@gvanrossum gvanrossum Oct 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't someone advise last time not to combine error checks like this?

Copy link
Copy Markdown
Member Author

@iritkatriel iritkatriel Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't someone advise last time not to combine error checks like this?

Yes, about PyList_Append calls which can crash. Here and in pythonrun.c (in main) the code seems to assume that PyFile_WriteXX calls can be called any number of times after failure (is this true?). There is a PyErr_Clear() call before each PyFile_WriteObject because that asserts that there is no error set, but the call is not avoided.

There is another PyErr_Clear at the end (these function don't have a return value and don't raise exceptions).

See this comment

cpython/Python/pythonrun.c

Line 955 in 19a6c41

/* Can't be bothered to check all those

Copy link
Copy Markdown
Member

@gvanrossum gvanrossum Oct 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More specifically, there are calls to PyErr_occurred() in PyFile_WriteString() that are clearly designed to make this safe. But AFAICT only there, not in PyFile_WriteObject().

Copy link
Copy Markdown
Member Author

@iritkatriel iritkatriel Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a mess, so how about I make a PR to clean up the current code's error checking, we merge that, then I recreate this one.

In the meantime we can sort out the last decisions:

(1) Do we want the "full path" labels? (I think they don't add that much on top of index + indent, and they complicate the code. Let me know if you disagree.)

(2) The box around the topmost EG's traceback?

(3) We definitely need to limit the nested depth we display (to something like 10 or 15). Also the number of exceptions per EG, so we don't spam?

Copy link
Copy Markdown
Member

@1st1 1st1 Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my 2c:

(2) The box around the topmost EG's traceback?

#29207 (comment)

(3) We definitely need to limit the nested depth we display (to something like 10 or 15). Also the number of exceptions per EG, so we don't spam?

I think it's a great idea. Perhaps we should have an env var to set the limits.

Copy link
Copy Markdown
Member

@gvanrossum gvanrossum Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a mess, so how about I make a PR to clean up the current code's error checking, we merge that, then I recreate this one.

Why not keep this PR but merge from main once the cleanup PR has landed?

Copy link
Copy Markdown
Member Author

@iritkatriel iritkatriel Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only a limit on the number of traceback frames, which applies to chained exceptions too.
It seems to be 1000 by default in C:

cpython/Python/traceback.c

Line 881 in d02ffd1

#define PyTraceBack_LIMIT 1000

and 'unlimited' in python:

cpython/Lib/traceback.py

Line 377 in 10bbd41

:param limit: None to include all frames or the number of frames to

There is no limit on the number of chained exceptions being printed (which is why we needed the loop to avoid recursion in traceback.py).

Copy link
Copy Markdown
Member

@gvanrossum gvanrossum Oct 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no limit on the number of chained exceptions being printed (which is why we needed the loop to avoid recursion in traceback.py).

That sounds like a design bug. That limit has been around since long before we had chaining; presumably nobody thought of adding a limit. Unlimited output seems unhelpful. A parameter that limits things seems useful though.

Copy link
Copy Markdown
Member Author

@iritkatriel iritkatriel Nov 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a mess, so how about I make a PR to clean up the current code's error checking, we merge that, then I recreate this one.

Change of plans - I found that its more delicate than I originally thought (see bpo-45635). I don't want to hold up this PR until that's sorted, so I'll make sure I'm not adding more error checking issues in the code I'm adding here, but leave the rest for later.

Copy link
Copy Markdown
Member Author

@iritkatriel iritkatriel Nov 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added two limits max_group_depth and max_group_width. If we roll everything into one limit I think we will get a constant stream of bug reports about edge cases where you got the wrong part of the output. Though it would probably be rare that it makes a difference in practice.

I didn't add a limit on the length of context/cause chain, because that's outside the scope of this PR. I created bpo-45694 for that.

Expand Down Expand Up @@ -842,9 +850,11 @@ tb_displayline(PyTracebackObject* tb, PyObject *f, PyObject *filename, int linen
}

err = _Py_WriteIndentedMargin(margin_indent, margin, f);
err |= print_error_location_carets(f, truncation, start_offset, end_offset,
right_start_offset, left_end_offset,
primary_error_char, secondary_error_char);
if (err == 0) {
err = print_error_location_carets(f, truncation, start_offset, end_offset,
right_start_offset, left_end_offset,
primary_error_char, secondary_error_char);
}

done:
Py_XDECREF(source_line);
Expand Down Expand Up @@ -948,8 +958,10 @@ _PyTraceBack_Print_Indented(PyObject *v, int indent, const char *margin,
}
}
err = _Py_WriteIndentedMargin(indent, header_margin, f);
err |= PyFile_WriteString(header, f);
if (!err) {
if (err == 0) {
err = PyFile_WriteString(header, f);
}
if (err == 0) {
err = tb_printinternal((PyTracebackObject *)v, f, limit, indent, margin);
Comment thread
iritkatriel marked this conversation as resolved.
}
return err;
Expand Down