bpo-36389: Add _PyObject_CheckConsistency() function #12803

vstinner · 2019-04-12T13:51:41Z

Add a new _PyObject_CheckConsistency() function which can be used to
help debugging. The function is available in release mode.

Add a 'check_content' parameter to _PyDict_CheckConsistency().

https://bugs.python.org/issue36389

Add a new _PyObject_CheckConsistency() function which can be used to help debugging. The function is available in release mode. Add a 'check_content' parameter to _PyDict_CheckConsistency().

zooba · 2019-04-12T15:39:22Z

Include/cpython/object.h

+   If check_content is zero, only check header fields: reduce the overhead.
+
+   The function always return 1. The return value is just here to be able to
+   write:


Why would you write assert(_PyObject_CheckConsistency) if it always returns 1? Wouldn't it be better off being named AssertConsistent if it's going to do the assert itself?

It looks like the other functions like this return 1 or 0 and the caller does the assert.

Why would you write assert(_PyObject_CheckConsistency) if it always returns 1?

It's a practical solution to add checks in debug mode which are omitted in release mode. It's shorter than:

#ifndef NDEBUG _PyObject_CheckConsistency(); #endif

or

#ifdef Py_DEBUG _PyObject_CheckConsistency(); #endif

By the way, I recall that someone reported an issue when Py_DEBUG is build without assertions (with NDEBUG defined).

Ah, got it, that makes sense. But if this is internal, why not just offer a macro that is defined to blank?

On Windows, Py_DEBUG is inferred from _DEBUG, which is the opposite of NDEBUG by default (but I'm not totally sure which one modifies assert). We should definitely use Py_DEBUG within the codebase for our own checks like this, and probably people should not set it directly? Or maybe we need to add some extra validation in PC/pyconfig.h to fail the build if it's inconsistent. (Let's worry about this when someone files a bug, rather than here :) )

zooba · 2019-04-12T15:40:37Z

Include/internal/pycore_object.h

@@ -10,6 +10,10 @@ extern "C" {

 #include "pycore_pystate.h"   /* _PyRuntime */

+PyAPI_FUNC(int) _PyType_CheckConsistency(PyTypeObject *type);
+PyAPI_FUNC(int) _PyUnicode_CheckConsistency(PyObject *op, int check_content);
+PyAPI_FUNC(int) _PyDict_CheckConsistency(PyObject *mp, int check_content);


Can we have a PyTuple_CheckConsistency that checks all elements are non-null and haven't been deallocated? We know there's a bug like this right now somewhere (though whether checking via the tuple would detect it or not I don't know...)

vstinner · 2019-04-12T16:35:20Z

Can we have a PyTuple_CheckConsistency that checks all elements are non-null and haven't been deallocated? We know there's a bug like this right now somewhere (though whether checking via the tuple would detect it or not I don't know...)

Aha, good question. Python internals are fully of micro-optimizations using cached tuples which are "more or less consistent". I'm thinking as such "hack" which are causing a lot of issues with PYTHONDUMPREFS=1 is used:

static PyObject *
property_descr_get(PyObject *self, PyObject *obj, PyObject *type)
{
    static PyObject * volatile cached_args = NULL;
    ...
    args = cached_args;
    cached_args = NULL;
    if (!args) {
        args = PyTuple_New(1);
        if (!args)
            return NULL;
        _PyObject_GC_UNTRACK(args);
    }
    Py_INCREF(obj);
    PyTuple_SET_ITEM(args, 0, obj);
    ret = PyObject_Call(gs->prop_get, args, NULL);
    if (cached_args == NULL && Py_REFCNT(args) == 1) {
        assert(Py_SIZE(args) == 1);
        assert(PyTuple_GET_ITEM(args, 0) == obj);
        cached_args = args;
        Py_DECREF(obj);
    }
    else {
        assert(Py_REFCNT(args) >= 1);
        _PyObject_GC_TRACK(args);
        Py_DECREF(args);
    }
    return ret;
}

I worked for 1 or 2 years to implement METH_FASTCALL basically to get ride of such evil micro-optimizations which lives an inconsistent tuple supposed to be hidden, but seen by Py_TRACE_REFS magic list, or sometimes by the GC (property_descr_get() has been "fixed" 2 or 3 times...).

Would you be interested to work on a PR written on top of this one, once I merge this PR?

zooba · 2019-04-12T17:36:34Z

Would you be interested to work on a PR written on top of this one, once I merge this PR?

You make it sound pretty scary :) But that cache in the code you quoted is also pretty scary.

I'm going to drag @ericsnowcurrently in here, since these kinds of things are going to cause havoc with subinterpreters.

vstinner · 2019-04-12T19:52:33Z

I merged my PR.

You make it sound pretty scary :) But that cache in the code you quoted is also pretty scary.

Hopefully, slowly, we are getting ride of these micro-optimizations ;-)

ZackerySpytz · 2019-04-12T20:39:07Z

Objects/object.c

+        fprintf(stderr, "<object: ob_type=NULL>\n");
+    }
+    else if (_PyObject_IsFreed((PyObject *)Py_TYPE(obj))) {
+        fprintf(stderr, "<object: freed type %p>\n", Py_TYPE(obj));


@vstinner If the argument for %p is not a pointer to void, the behavior is undefined. There needs to be a (void *) cast just before the Py_TYPE(obj) (see also GH-12769).

Needs care because I haven't yet checked *every* place this is called. But the bulk of places it's called are in _PyUnicode_CheckConsistency and its friends -- where if NDEBUG we don't want to even call them. So, pending confirmation of that (and perhaps migrating any exceptions to a slightly different API), this should be just fine. Indeed we could go further and not even check _PyObject_ASSERTIONS here. Probably should use a different name from "assert" then, though. --- There's a warning when doing a release build: Objects/unicodeobject.c: In function ‘_PyUnicode_CheckConsistency’: Objects/unicodeobject.c:507:15: warning: variable ‘data’ set but not used [-Wunused-but-set-variable] void *data; ^~~~ That's with: $ gcc --version gcc (Debian 8.3.0-6) 8.3.0 Introduced here: commit 0fc91ee Author: Victor Stinner <[email protected]> Date: Fri Apr 12 21:51:34 2019 +0200 bpo-36389: Add _PyObject_CheckConsistency() function (pythonGH-12803) Add a new _PyObject_CheckConsistency() function which can be used to help debugging. The function is available in release mode. Add a 'check_content' parameter to _PyDict_CheckConsistency(). Include/cpython/object.h | 15 ++++++ Include/internal/pycore_object.h | 4 ++ Objects/dictobject.c | 118 +++++++++++++++++++++---------------------- Objects/object.c | 31 +++++++++++- Objects/typeobject.c | 11 ++-- Objects/unicodeobject.c | 93 ++++++++++++++++------------------ 6 files changed, 158 insertions(+), 114 deletions(-)

bpo-36389: Add _PyObject_CheckConsistency() function

77fd0e4

Add a new _PyObject_CheckConsistency() function which can be used to help debugging. The function is available in release mode. Add a 'check_content' parameter to _PyDict_CheckConsistency().

vstinner requested a review from methane as a code owner April 12, 2019 13:51

the-knights-who-say-ni added the CLA signed label Apr 12, 2019

bedevere-bot added the awaiting core review label Apr 12, 2019

zooba reviewed Apr 12, 2019

View reviewed changes

vstinner added the skip news label Apr 12, 2019

vstinner merged commit 0fc91ee into python:master Apr 12, 2019

bedevere-bot removed the awaiting core review label Apr 12, 2019

vstinner deleted the object_check_consistency branch April 12, 2019 19:51

ZackerySpytz reviewed Apr 12, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-36389: Add _PyObject_CheckConsistency() function #12803

bpo-36389: Add _PyObject_CheckConsistency() function #12803

vstinner commented Apr 12, 2019 •

edited by bedevere-bot

Loading

zooba Apr 12, 2019

vstinner Apr 12, 2019

zooba Apr 12, 2019

zooba Apr 12, 2019

vstinner commented Apr 12, 2019

zooba commented Apr 12, 2019

vstinner commented Apr 12, 2019

ZackerySpytz Apr 12, 2019

bpo-36389: Add _PyObject_CheckConsistency() function #12803

bpo-36389: Add _PyObject_CheckConsistency() function #12803

Conversation

vstinner commented Apr 12, 2019 • edited by bedevere-bot Loading

zooba Apr 12, 2019

Choose a reason for hiding this comment

vstinner Apr 12, 2019

Choose a reason for hiding this comment

zooba Apr 12, 2019

Choose a reason for hiding this comment

zooba Apr 12, 2019

Choose a reason for hiding this comment

vstinner commented Apr 12, 2019

zooba commented Apr 12, 2019

vstinner commented Apr 12, 2019

ZackerySpytz Apr 12, 2019

Choose a reason for hiding this comment

vstinner commented Apr 12, 2019 •

edited by bedevere-bot

Loading