Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
1b01a91
Add `f_generator` property to Python frame objects
1st1 Sep 24, 2024
0fc5511
Working implementation of `asyncio.capture_call_stack()`
1st1 Sep 25, 2024
1d20a51
Address Guido's comments
1st1 Sep 26, 2024
c8be18e
Add a comment for capture_call_stack()
1st1 Sep 26, 2024
abf2cb9
Add a couple more tests
1st1 Sep 26, 2024
20ceab7
Remove setter for C impl of Task._awaited_by
1st1 Sep 26, 2024
72d9321
Intoduce cr_task
1st1 Sep 26, 2024
c9475f6
Unbreak shield() and gather()
1st1 Sep 26, 2024
e1099e9
Add convinience fields to C Task/Future for profilers
1st1 Sep 26, 2024
817f88b
Fix ups
1st1 Sep 27, 2024
54386ac
Add basic docs
1st1 Sep 27, 2024
98434f0
Implement, test, and document asyncio.print_call_stack()
1st1 Sep 28, 2024
485c166
Reorder a few things
1st1 Sep 28, 2024
8802be7
Add a test to exercise asyncio stack traces in out-of-process profilers
pablogsal Sep 29, 2024
1ddc9cf
Refactor the section-generated code into an evil macro
pablogsal Sep 30, 2024
bc9beb8
Rename "capture_call_stack()" et al to "capture_call_graph()"
1st1 Oct 2, 2024
fd141d4
Add NEWS
1st1 Oct 2, 2024
2d72f24
Per mpage's suggestion: use traceback style formatring for frames
1st1 Oct 2, 2024
391defa
mpage feedback: fix typo
1st1 Oct 2, 2024
d6357fd
Per mpage suggestion: get rid of CoroutineCallGraphEntry
1st1 Oct 2, 2024
bb3b6df
Strip sloppy white space!
1st1 Oct 2, 2024
54c99ec
Fix sloppy set iteration!
1st1 Oct 2, 2024
c1a4f09
Fix a sloppy test!
1st1 Oct 2, 2024
027d522
Run 'make regen-all'
1st1 Oct 2, 2024
c2d5ec6
Add a what's new entry
1st1 Oct 2, 2024
08d09eb
Fix (hopefully) a compiler warning
1st1 Oct 2, 2024
fe3113b
Fix sloppy what's new!
1st1 Oct 2, 2024
18ec26d
Fix CI complaining about suspicious global in C
1st1 Oct 2, 2024
e4cc462
Add a test for depth=2
1st1 Oct 2, 2024
d5cdc36
Add critical sections for free threaded builds
pablogsal Oct 2, 2024
83606f2
Add more slopy tests
pablogsal Oct 2, 2024
5edac41
Apply suggestions from code review
1st1 Oct 16, 2024
8dc6d34
Apply suggestions from code review
1st1 Oct 16, 2024
30884ea
Update Modules/_asynciomodule.c
1st1 Oct 16, 2024
1317658
Update Modules/_asynciomodule.c
1st1 Oct 16, 2024
81b0a31
Use dataclasses instead of tuples for asyncio.stack
1st1 Oct 16, 2024
258ce3d
Fix sloppy whitespace!
1st1 Oct 16, 2024
b9ecefb
Remove critical sections for now
pablogsal Oct 16, 2024
b77dcb0
Eplain the weird macro
1st1 Oct 16, 2024
8867946
Fix test
pablogsal Oct 16, 2024
87d2524
Merge remote-tracking branch 'upstream/main' into stack
pablogsal Oct 16, 2024
b47bef1
Add a docs clarification
1st1 Oct 16, 2024
230b7ec
Fix typing in asyncio.stack
ambv Oct 16, 2024
b1d6158
Fix memory leak
pablogsal Oct 17, 2024
ac51364
Address comments from Kumar's review
ambv Oct 17, 2024
c7e59eb
Fix rare Mach-O linker bug when .bss is uninitialized data
ambv Oct 22, 2024
9eba5e1
Merge branch 'main' into pr-124640
ambv Oct 22, 2024
59121f6
you saw nothing, okay
ambv Oct 22, 2024
f8f48f0
Update Lib/asyncio/futures.py
1st1 Oct 23, 2024
74c5ad1
Remove unnecessary imports from tests
ambv Oct 30, 2024
067c043
Remove cr_task/gi_task
ambv Oct 31, 2024
9f04911
Fix refleaks
ambv Nov 13, 2024
0774805
Rename "asyncio.stack" to "asyncio.graph"
ambv Nov 13, 2024
3048493
Allow returning a string with `format_call_graph`
ambv Nov 13, 2024
1f42873
Add test for eager task factory support
ambv Nov 13, 2024
7799391
Merge branch 'main' into pr-124640
ambv Nov 13, 2024
03ed5c1
Make _asyncio_awaited_by a frozenset in the Python version as well pe…
ambv Nov 13, 2024
21f9ea9
Address picnixz' feedback
1st1 Nov 23, 2024
8a43dfa
Blacken the C/Py test code by hand, by request from picnixz
1st1 Nov 23, 2024
b3fae68
More style fixes per picnixz' suggestions
1st1 Nov 23, 2024
d0aedf0
Address Kumar's latest comment
1st1 Nov 23, 2024
df0032a
diff cleanup
kumaraditya303 Nov 23, 2024
0ce241b
change dataclasses to use tuples
kumaraditya303 Nov 23, 2024
8f126f6
doc fixes
kumaraditya303 Nov 23, 2024
966d84e
remove reduntant headers include and add my name to whatsnew
kumaraditya303 Nov 23, 2024
f56468a
improve comment
kumaraditya303 Nov 23, 2024
404b88a
fix leave task
kumaraditya303 Nov 23, 2024
911fed8
fix external inspection on linux
kumaraditya303 Nov 23, 2024
ab511a4
minor format
kumaraditya303 Nov 23, 2024
c3c685a
try to fix docs build
kumaraditya303 Nov 23, 2024
785adeb
Update Doc/whatsnew/3.14.rst
kumaraditya303 Nov 23, 2024
a577328
Match indentation with _asynciomodule.c after ab511a4f67ffd88f7709c1b…
ambv Nov 23, 2024
064129a
Improve comments after f56468af8b3789d0e20be30715c20ac1bb367641
ambv Nov 23, 2024
ce332d9
Restore the Oxford comma
ambv Nov 23, 2024
d6d943f
Address remaining suggestions of Andrew Svetlov
ambv Nov 23, 2024
703ff46
Fix gather()
1st1 Nov 23, 2024
e867863
Replace lambda by closure
pablogsal Nov 26, 2024
9cb5b29
Let’s not emphasize *asyncio*
1st1 Nov 26, 2024
61b2b7b
Apply suggestions from Irit's review
ambv Nov 26, 2024
9533ab9
Address remaining review by Irit
ambv Nov 26, 2024
596191d
Test pure-Python and C-accelerated versions of future_add_to/future_d…
ambv Nov 26, 2024
ad9152e
fix blooper
ambv Nov 26, 2024
066bf21
Fix another blooper
ambv Nov 26, 2024
4caeec4
Fix crash
pablogsal Jan 21, 2025
38f061d
Merge remote-tracking branch 'upstream/main' into stack
pablogsal Jan 21, 2025
a8dd667
use private method for the policy
pablogsal Jan 22, 2025
cf8f5e5
Avoid importing typing
ambv Jan 22, 2025
eda9c7c
Remove debug printing from test_asyncio.test_graph
ambv Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove cr_task/gi_task
  • Loading branch information
ambv committed Nov 13, 2024
commit 067c043cc4792fa19a04a905b16a2da3731a7ece
2 changes: 0 additions & 2 deletions Include/cpython/genobject.h
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,6 @@ PyAPI_DATA(PyTypeObject) PyCoro_Type;
PyAPI_FUNC(PyObject *) PyCoro_New(PyFrameObject *,
PyObject *name, PyObject *qualname);

PyAPI_FUNC(void) _PyCoro_SetTask(PyObject *coro, PyObject *task);


/* --- Asynchronous Generators -------------------------------------------- */

Expand Down
2 changes: 0 additions & 2 deletions Include/internal/pycore_debug_offsets.h
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,6 @@ typedef struct _Py_DebugOffsets {
uint64_t size;
uint64_t gi_name;
uint64_t gi_iframe;
uint64_t gi_task;
uint64_t gi_frame_state;
} gen_object;
} _Py_DebugOffsets;
Expand Down Expand Up @@ -324,7 +323,6 @@ typedef struct _Py_DebugOffsets {
.size = sizeof(PyGenObject), \
.gi_name = offsetof(PyGenObject, gi_name), \
.gi_iframe = offsetof(PyGenObject, gi_iframe), \
.gi_task = offsetof(PyGenObject, gi_task), \
.gi_frame_state = offsetof(PyGenObject, gi_frame_state), \
}, \
}
Expand Down
7 changes: 0 additions & 7 deletions Include/internal/pycore_genobject.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,6 @@ extern "C" {
PyObject *prefix##_qualname; \
_PyErr_StackItem prefix##_exc_state; \
PyObject *prefix##_origin_or_finalizer; \
/* A *borrowed* reference to a task that drives the coroutine. \
The field is meant to be used by profilers and debuggers only. \
The main invariant is that a task can't get GC'ed while \
the coroutine it drives is alive and vice versa. \
Profilers can use this field to reconstruct the full async \
call stack of program. */ \
PyObject *prefix##_task; \
char prefix##_hooks_inited; \
char prefix##_closed; \
char prefix##_running_async; \
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_tstate.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ typedef struct _PyThreadStateImpl {
PyThreadState base;

PyObject *asyncio_running_loop; // Strong reference
PyObject *asyncio_running_task; // Strong reference

struct _qsbr_thread_state *qsbr; // only used by free-threaded build
struct llist_node mem_free_queue; // delayed free queue
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_sys.py
Original file line number Diff line number Diff line change
Expand Up @@ -1617,7 +1617,7 @@ def bar(cls):
check(bar, size('PP'))
# generator
def get_gen(): yield 1
check(get_gen(), size('7P4c' + INTERPRETER_FRAME + 'P'))
check(get_gen(), size('6P4c' + INTERPRETER_FRAME + 'P'))
# iterator
check(iter('abc'), size('lP'))
# callable-iterator
Expand Down
32 changes: 20 additions & 12 deletions Modules/_asynciomodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,6 @@ typedef struct {
PyObject *sw_arg;
} TaskStepMethWrapper;


#define Future_CheckExact(state, obj) Py_IS_TYPE(obj, state->FutureType)
#define Task_CheckExact(state, obj) Py_IS_TYPE(obj, state->TaskType)

Expand Down Expand Up @@ -113,6 +112,11 @@ typedef struct _Py_AsyncioModuleDebugOffsets {
uint64_t task_awaited_by_is_set;
uint64_t task_coro;
} asyncio_task_object;
struct _asyncio_thread_state {
uint64_t size;
uint64_t asyncio_running_loop;
uint64_t asyncio_running_task;
} asyncio_thread_state;
} Py_AsyncioModuleDebugOffsets;

GENERATE_DEBUG_SECTION(AsyncioDebug, Py_AsyncioModuleDebugOffsets AsyncioDebug)
Comment thread
1st1 marked this conversation as resolved.
Expand All @@ -123,6 +127,11 @@ GENERATE_DEBUG_SECTION(AsyncioDebug, Py_AsyncioModuleDebugOffsets AsyncioDebug)
.task_is_task = offsetof(TaskObj, task_is_task),
.task_awaited_by_is_set = offsetof(TaskObj, task_awaited_by_is_set),
.task_coro = offsetof(TaskObj, task_coro),
},
.asyncio_thread_state = {
.size = sizeof(_PyThreadStateImpl),
.asyncio_running_loop = offsetof(_PyThreadStateImpl, asyncio_running_loop),
.asyncio_running_task = offsetof(_PyThreadStateImpl, asyncio_running_task),
}};

/* State of the _asyncio module */
Expand Down Expand Up @@ -219,7 +228,6 @@ typedef struct {
TaskObj tail;
TaskObj *head;
} asyncio_tasks;

} asyncio_state;

static inline asyncio_state *
Expand Down Expand Up @@ -268,9 +276,6 @@ task_step_handle_result_impl(asyncio_state *state, TaskObj *task, PyObject *resu
static void
clear_task_coro(TaskObj *task)
{
if (task->task_coro != NULL && PyCoro_CheckExact(task->task_coro)) {
_PyCoro_SetTask(task->task_coro, NULL);
}
Py_CLEAR(task->task_coro);
}

Expand All @@ -279,9 +284,6 @@ static void
set_task_coro(TaskObj *task, PyObject *coro)
{
assert(coro != NULL);
if (PyCoro_CheckExact(coro)) {
_PyCoro_SetTask(coro, (PyObject *)task);
}
Py_INCREF(coro);
Py_XSETREF(task->task_coro, coro);
}
Expand Down Expand Up @@ -2160,7 +2162,10 @@ enter_task(asyncio_state *state, PyObject *loop, PyObject *task)
Py_DECREF(item);
return -1;
}
Py_DECREF(item);

_PyThreadStateImpl *ts = (_PyThreadStateImpl *)_PyThreadState_GET();
assert(ts->asyncio_running_task == NULL);
ts->asyncio_running_task = item; // strong ref
Copy link
Copy Markdown
Contributor

@kumaraditya303 kumaraditya303 Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't safe. You can only do this when you can guarantee that running loop is the loop of this task. This function can be called from any thread.

Copy link
Copy Markdown
Contributor

@kumaraditya303 kumaraditya303 Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea which would work on all cases:

  • Remove this tracking of current task tstate
  • put the current task on interpreter state
  • expose the offset of those and read it interp state.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just expose the current loop and let them read the current task from the current_tasks dict above, the dict offsets are already exposed.

Because reading a dictionary is very complex externally. You cannot run Python code, you'd have to recreate the implementation on the other end, with all the pointer indirection.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kumar, this is a very good catch.

I agree that additional checks are warranted here. But I disagree that the current approach overall isn't valid. While technically "enter_task" can be called from some unrelated thread, in practice, it will only be called from one place -- around the Task's stepping function. For our asyncio Task that will always be the case, for third-party tasks that don't inherit from asyncio.Task I have to comments:

  • we don't support tracking of third-party tasks with the new API anyway, so we just need to make sure we don't crash if a rogue task does highly questionable things;
  • I can't imagine why would anyone on earth would call "asyncio.enter_task" from a different thread that their task is running in. Like it's just hard to even implement that, left alone have a sane motivation for that extra work. Or am I missing something? Please correct me if I'm wrong.

Therefore I'd argue that the current approach is OK, but I'm adding an additional check to the logic as well as an extensive comment elaborating on what's going on.

CleanShot 2024-11-22 at 20 54 51@2x

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't imagine why would anyone on earth would call "asyncio.enter_task" from a different thread that their task is running in. Like it's just hard to even implement that, left alone have a sane motivation for that extra work. Or am I missing something? Please correct me if I'm wrong.

This may happen if in case someone is running two separate event loops in two threads and tries to introspect the call stack of the other. I understand that this may seem far fetch but still better safe than sorry. Pushed a minor suggestion to your comment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may happen if in case someone is running two separate event loops in two threads and tries to introspect the call stack of the other. I understand that this may seem far fetch but still better safe than sorry. Pushed a minor suggestion to your comment.

That will work, introspection doesn't rely on enter_task itself. So, again, I don't see how what you're saying is possible at all.

return 0;
}

Expand All @@ -2185,14 +2190,16 @@ leave_task_predicate(PyObject *item, void *task)

static int
leave_task(asyncio_state *state, PyObject *loop, PyObject *task)
/*[clinic end generated code: output=0ebf6db4b858fb41 input=51296a46313d1ad8]*/
{
int res = _PyDict_DelItemIf(state->current_tasks, loop,
leave_task_predicate, task);
if (res == 0) {
// task was not found
return err_leave_task(Py_None, task);
}

_PyThreadStateImpl *ts = (_PyThreadStateImpl *)_PyThreadState_GET();
Py_CLEAR(ts->asyncio_running_task);
return res;
}

Expand Down Expand Up @@ -3960,7 +3967,9 @@ module_clear(PyObject *mod)
Py_CLEAR(state->iscoroutine_typecache);

Py_CLEAR(state->context_kwname);

_PyThreadStateImpl *ts = (_PyThreadStateImpl *)_PyThreadState_GET();
Py_CLEAR(ts->asyncio_running_loop);
Py_CLEAR(ts->asyncio_running_task);
return 0;
}

Expand Down Expand Up @@ -3990,7 +3999,6 @@ module_init(asyncio_state *state)
goto fail;
}


state->context_kwname = Py_BuildValue("(s)", "context");
if (state->context_kwname == NULL) {
goto fail;
Expand Down
Loading