Right, it is out of scope but it is an easy CPU bomb: in my case, a 1x5 image took 3 minutes to process because it had entries like 1e40.

All those while loops do is get the remainder of a division by 6. We might as well use the official function for that (and transform the while loop into conditions). The SIMD implementation does that too.

@vrabaud, thank you for the patch! I agree with both you and @alalek, there should be protection against attacks or unintentional bad usage, but it should preferably not slowdown the normal use cases. fmodf means function call, and it can be quite heavy, as it may include some checks for corner cases etc.

May I suggest to modify the patch as:

sector = cvFloor(h);
h -= sector; 
sector %= 6;
sector += sector < 0 ? 6 : 0;

/* ///////// these instructions are not needed anymore,
   we guarantee that sector is within [0, 6) //////////
CV_DbgAssert( 0 <= h && h < 6 );
sector = cvFloor(h);
h -= sector;
*/

I found that the generated code is very good, much better than what we had before. sector %= 6 is computed efficiently without divisions.

Thx @vpisarev , I applied your patch and also added a fix so that NaN does not trigger out of bound reads.

We actually have another problem for big h values: cvFloor will SIGILL
We can do:

                h *= hscale;
                if (cvIsNaN(h)) {
                    // Avoid throwing cvFloor computation in the NaN case.
                    b = g = r = h;
                } else {
                    if  (h > std::numeric_limits<int>::max() ||
                         h < std::numeric_limits<int>::min(){
                       // standard C++ floor
                    } else {
                      cvFloor
                    }

@vrabaud, this problem with cvFloor() is quite serious actually. What is the platform (Intel, ARM, ???), OS and the compiler where this problem is reproduced? What are the particular values that you pass to cvFloor()?

@vrabaud friendly reminder.

Ok, I updated my PR

• unify the sector computation because we have the same issue with HSV
• the SIGILL was coming from the fuzzer compilation flag because a float bigger than numeric_limits::max() was cast to int in cvFloor (using the C cast not __builtin_floof)
• fmodf has to be used first in case we have a float bigger that the max int
• the NaN case is treated independently to not trigger sector values out of memory (which would be a security issue)

We can have a second path if h is smaller than max int like what @vpisarev mentioned but it is just replacing an fmod by an %, not much of a gain.

@vrabaud, @asmorkalov, the solution is robust, but I don't like that it seriously affects the speed in normal cases, compared to the solution that I suggested. Also, the solution is very local to HLS2RGB, it does not solve the problem of SIGILL when cvFloor() is given NaN or +/-Inf or another very big value. I suggest to solve the problem once and for all:

1. We declare in documentation that cvFloor() (as well as cvCeil, cvRound) produces some platform-dependent integer value when it's given a NaN or argument that is outside of INT_MIN..INT_MAX.
2. We make sure that if result of cvFloor() (cvCeil, cvRound) is used to compute address or index in LUT, it does not go outside of the proper value range.
3. We make sure that implementation of cvFloor() (cvCeil, cvRound) is very efficient and that it does not throw any exception when the argument is NaN or is outside of INT_MIN..INT_MAX.

First, we need to see if __builtin_floorf (__builtin_ceilf, __builtin_lrintf) produce exception. If not, let's use them for cvFloor/cvCeil/cvRound implementation. This is what the current 4.x branch does. If not, let's check if (int)x produces exception. If not, let's change implementation of cvFloor(), cvCeil() back to what it was before the latest patches:

int cvFloorf/*_exception_less*/(float x) {
    int i = (int)x;
    return i - (i > x);
}

int cvCeilf/*_exception_less*/(float x) {
    int i = (int)x;
    return i + (i < x);
}

Sorry for the delay, I applied @vpisarev 's suggestion and removed the NaN handling case (the throwing was just due to the sanitizer throwing when shrinking doing a float to int cast).

@alalek, should we finally merge it?