Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[Windows Python 3.5 only] np.clip replace nans with lower bound #7601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lesteve opened this issue May 4, 2016 · 12 comments · Fixed by #7678
Closed

[Windows Python 3.5 only] np.clip replace nans with lower bound #7601

lesteve opened this issue May 4, 2016 · 12 comments · Fixed by #7678

Comments

@lesteve
Copy link
Contributor

lesteve commented May 4, 2016

import numpy as np

arr = np.array([np.nan, 10])
print(np.clip(arr, -1, 1))

Expected output is that nan are left untouched:

[ nan   1.]

Actual output with Python 3.5 on a Windows VM (numpy 1.11 installed through conda):

[-1.  1.]

It works as expected with Python 2.7 and 3.4 on the Windows VM. It works as expected with all Python versions on my Ubuntu box.

@matthew-brett
Copy link
Contributor

Sorry, I can't test at the moment (no access to a Windows machine or VM) but would you mind testing with the current numpy wheel on pypi? I imagine it has the same error, because it's built with the same compiler, but it would be good to be sure. Do you get the same error for 32- and 64-bit?

@lesteve
Copy link
Contributor Author

lesteve commented May 4, 2016

This is 64 bit, I haven't tested with 32-bit.

I get the same behaviour with the numpy wheel from PyPI.

@lesteve
Copy link
Contributor Author

lesteve commented May 10, 2016

I started looking at it a bit further and I see a very surprising behaviour. It seems like adding a printf statement inside the filling loop in numpy/core/src/multiarray/arraytypes.c.src
somehow fixes it. If someone has any clue on how this is even remotely possible, I'd be very very interested!

I see this behaviour on my laptop and I reproduced it on AppVeyor:

@seberg
Copy link
Member

seberg commented May 10, 2016

I don't know this stuff well, but I would guess that the printf statement kills the compilers optimization to vectorized/SIMD instructions, which in turn changes the floating point error status setting stuff. If you compile with -O0 it probably always works.
Anyway, I don't know the depth of SSE, non-vectorized, etc. floating point error flag setting, especially on windows, but maybe someone else will know it.

@matthew-brett
Copy link
Contributor

matthew-brett commented May 10, 2016 via email

@lesteve
Copy link
Contributor Author

lesteve commented May 10, 2016

Loic - this is rather a serious bug - would you consider writing a failing test case for it to put into the numpy test suite?

There is one in my branch: lesteve@2c257cf

The AppVeyor failure can be seen here.

Or did you mean something else?

@lesteve
Copy link
Contributor Author

lesteve commented May 12, 2016

I managed to reproduce the problem in a stand-alone C program.

#include <stdio.h>
#include <math.h>
#include <stdlib.h>

#define SIZE 2

void clip(double* in, int size, double min, double max, double* out){
  for (int i=0; i < size; i++){
    if (in[i] < min){
      out[i] = min;
    }
    else if (in[i] > max) {
      out[i] = max;
    }
    else {
      out[i] = in[i];
    }
  }
}

int main() {
  double* in = malloc(sizeof(double) * SIZE);
  double* out = malloc(sizeof(double) * SIZE);
  double min = -1.;
  double max = 1.;

  in[0] = NAN;
  in[1] = 10;

  clip(in, SIZE, min, max, out);

  for (int i=0; i < SIZE; i++){
      printf("i: %d, value: %f\n", i, out[i]);
    }
}

As @seberg it has something to do with optimizations. Compiling it with cl clip_bug.c (without optimization) and running it works as expected with the output:

i: 0, value: -nan(ind)
i: 1, value: 1.000000

Compiling it with cl /Ox clip_bug.c (maximum optimization, which numpy uses AFAICS) reproduces the bug with the output:

i: 0, value: -1.000000
i: 1, value: 1.000000

Does anyone have any advice on where to report this bug?

@pv
Copy link
Member

pv commented May 12, 2016 via email

@lesteve
Copy link
Contributor Author

lesteve commented May 19, 2016

I just opened a ticket at connect.microsoft.com here.

@eric-wieser
Copy link
Member

@lesteve: Any idea where that ticket moved to? The link is now dead, and I think this is causing #12520

@mattip
Copy link
Member

mattip commented Dec 10, 2018

@eric-wieser does that stand alone program fail in your setup?

@lesteve
Copy link
Contributor Author

lesteve commented Dec 10, 2018

@lesteve: Any idea where that ticket moved to?

It seems Microsoft Connect has been retired ... I could not find anything from googling and I could not find any archive online. From what I remember there was not much content in the Microsoft Connect ticket besides an acknowledgement that there was a bug and that the bug will be fixed in a further version of Visual Studio (I asked but the answer was not clear as to which version it was going to be exactly).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants