[MRG + 1] FIX gil acquisition in dist_metric #9311

glemaitre · 2017-07-10T05:54:48Z

Reference Issue

partly addressing #9272
continues #9133
continues #9296

What does this implement/fix? Explain your changes.

Due to the inheritance, dist needs to be declared with nogil.
Therefore a _dist declared with gil is used and called in dist.

Any other comments?

glemaitre · 2017-07-10T05:55:10Z

sklearn/neighbors/dist_metrics.pyx

@@ -1093,22 +1093,29 @@ cdef class PyFuncDistance(DistanceMetric):
        self.func = func
        self.kwargs = kwargs

+    # in cython < 0.26, GIL was required to be acquired during definition of


Do we need this comments or shall I remove it?

It's helpful

I don't understand why we needed to acquire it twice previously. Seems strange. But I've not been following the discussions.

There has been some cleanup regarding inconsistencies between nogil and withgil on declaration (including inheritance) vs. definition in previous versions of Cython. This seems like a fine workaround to support both versions.

glemaitre · 2017-07-10T05:57:25Z

@amueller @lesteve

The test were passing locally in both cython 0.25 and master.
I put a comment explaining the reason for this warper but I would appreciate if you could revise it (even remove it if we don't need it)

glemaitre · 2017-07-10T06:03:51Z

I forgot to ping @jnothman @ogrisel

jnothman

I don't understand. I'll try read the back-story when I have time.

jnothman · 2017-07-10T06:50:17Z

sklearn/neighbors/dist_metrics.pyx

+    # in cython < 0.26, GIL was required to be acquired during definition of
+    # the function and inside the body of the function. This behaviour is not
+    # allowed in cython >= 0.26 since it is a redundant GIL acquisition. The
+    # only way to be back compatible is to inherate `dist` from the base class


jnothman · 2017-07-10T06:50:24Z

sklearn/neighbors/dist_metrics.pyx

@@ -1093,22 +1093,29 @@ cdef class PyFuncDistance(DistanceMetric):
        self.func = func
        self.kwargs = kwargs

+    # in cython < 0.26, GIL was required to be acquired during definition of


It's helpful

jnothman · 2017-07-10T06:52:31Z

sklearn/neighbors/dist_metrics.pyx

@@ -1093,22 +1093,29 @@ cdef class PyFuncDistance(DistanceMetric):
        self.func = func
        self.kwargs = kwargs

+    # in cython < 0.26, GIL was required to be acquired during definition of


I don't understand why we needed to acquire it twice previously. Seems strange. But I've not been following the discussions.

jnothman · 2017-07-10T06:53:01Z

sklearn/neighbors/dist_metrics.pyx

    cdef inline DTYPE_t dist(self, DTYPE_t* x1, DTYPE_t* x2,
-                             ITYPE_t size) except -1 with gil:
+                             ITYPE_t size) nogil except -1:


I don't get why this should be no gil.

glemaitre · 2017-07-10T07:05:47Z

‎It seems like a Cython bug. My Cython skills are to weak to know the explanation. I could think about something like: self was kinda of inspected and func was found‎ to be a python object. Therefore dist needed to be declared/defined with gil but the inheritance (with nogil) was messing up, imposing to acquire the gil in the body.

TomDLT · 2017-07-10T08:33:52Z

Not a cython expert, but it LGTM

jmschrei · 2017-07-10T15:39:17Z

This looks good to me. I think the issue may have been that defining a function as nogil doesn't release the GIL, it simply allows the function to be called when the GIL is released, so the author may have thought that with gil worked the same way.

scoder · 2017-07-10T18:56:58Z

The explanation, IIUC, is that Cython requires the with gil signature declaration in order to allow local Python object variables, but previous Cython versions ignored it when it came to actually acquiring the GIL and instead looked at the nogil declaration of the base type method and did nothing at runtime. Thus, a second with gil block inside of the function body was required in order to acquire the GIL. This has been fixed and the duplicate GIL acquiring was turned into an error, but the change in this pull request is needed in order to support both the old buggy Cython versions and the new corrected ones.

amueller · 2017-07-10T19:41:52Z

LGTM.

Does this run on cython master? The beta segfaults, right?

glemaitre · 2017-07-10T19:52:26Z

The beta was segfaulting. But it was good on master.

TomDLT · 2017-07-18T11:26:09Z

The problem seems well understood, let's merge this fix

TomDLT · 2017-07-18T11:26:41Z

Thanks @glemaitre !

amueller · 2017-07-18T15:16:48Z

Thanks!

acquire the GIL in a new inline function

95f1bb0

glemaitre commented Jul 10, 2017

View reviewed changes

glemaitre mentioned this pull request Jul 10, 2017

build with cython 0.26 beta? #9272

Closed

jnothman reviewed Jul 10, 2017

View reviewed changes

fix typo

900ef92

amueller changed the title ~~[MRG] FIX gil acquisition in dist_metric~~ [MRG + 1] FIX gil acquisition in dist_metric Jul 10, 2017

TomDLT merged commit 8cee70d into scikit-learn:master Jul 18, 2017

TomDLT mentioned this pull request Jul 18, 2017

Remove redundant gil acquisition. #9133

Closed

amueller mentioned this pull request Jul 18, 2017

make dist_metrics.pyx compile with cython beta #9296

Closed

ogrisel pushed a commit that referenced this pull request Jul 24, 2017

[MRG + 1] FIX gil acquisition in dist_metric (#9311)

5c3db72

jnothman pushed a commit to jnothman/scikit-learn that referenced this pull request Aug 6, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

afa26da

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

7354380

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

55986df

paulha pushed a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

e7a8786

AishwaryaRK pushed a commit to AishwaryaRK/scikit-learn that referenced this pull request Aug 29, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

5efeb28

maskani-moh pushed a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

b0edfd7

RomainBrault mentioned this pull request Nov 27, 2017

Cythonize GIL error ngoix/OCRF#9

Open

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

[MRG + 1] FIX gil acquisition in dist_metric (scikit-learn#9311)

d95f0f0

Uh oh!

[MRG + 1] FIX gil acquisition in dist_metric #9311

[MRG + 1] FIX gil acquisition in dist_metric #9311

Uh oh!

Conversation

glemaitre commented Jul 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issue

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

glemaitre Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

robertwb Jul 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jul 10, 2017 • edited by TomDLT Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glemaitre commented Jul 10, 2017

Uh oh!

jnothman left a comment

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

jnothman Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jul 10, 2017 via email

Uh oh!

TomDLT commented Jul 10, 2017

Uh oh!

jmschrei commented Jul 10, 2017

Uh oh!

scoder commented Jul 10, 2017

Uh oh!

amueller commented Jul 10, 2017

Uh oh!

glemaitre commented Jul 10, 2017 via email • edited by amueller Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomDLT commented Jul 18, 2017

Uh oh!

TomDLT commented Jul 18, 2017

Uh oh!

amueller commented Jul 18, 2017

Uh oh!

Uh oh!

glemaitre commented Jul 10, 2017 •

edited

Loading

robertwb Jul 10, 2017 •

edited

Loading

glemaitre commented Jul 10, 2017 •

edited by TomDLT

Loading

glemaitre commented Jul 10, 2017 via email •

edited by amueller

Loading