awesome, thanks :)
Could it be that you forgot to add some files / modify the setup.py? Travis is complaining.

You didn't add the updates c file ;)

yep, I didn't want to clutter the commit log at this stage. I'll add some tests first and then care about this whiny Travis guy.

yep, I didn't want to clutter the commit log at this stage.

Please feel free to checkin the .c file as well. The diff clutter can be controlled by:

https://github.com/scikit-learn/scikit-learn/blob/master/.gitattributes

This looks like a great contrib @temporaer . Looking forward to the tests and some documentation updates in the preprocessing section.

@temporaer could you please squash the commits? (because that's something we do now ^^)

uh... squash /all/ of them into one? Or just clean up a little?

Preferably everything. That keeps the history slim, makes reverting easy, prevents broken states in the history, and hides our mistakes from the users ;)

Thanks. +1 for merge :)

@larsmans ok for merge?

The code looks alright, but I still don't understand the assume_contains_zeros option, and I want the API to be clear before merging this.

I agree. I'm also not sure what it is we actually want in the sparse case.
I think the default option should keep sparsity structure, as this is how we did it in other cases afaik (RandomizedPCA for example), and the user needs to be explicit to destroy sparsity structure.

So maybe we want an with_offset option which is True for dense and False for sparse input.

If the input if strictly positive, between 0 and x and the given range is [0, a] then the right thing will happen.
If the input is between -x and x and the range is [-a, a] also the right thing will happen.

In all other cases, something a bit weird would happen, though
If the input is between -x and x and you ask for [0, a] you would get [0, a/2] I think... well...

Making centering optional sounds good to me, it is somewhat more intuitive than "injecting" artificial zeros. We'd still need to explain the reasoning why the user would want to change default behavior, and there the complexity would be unchanged. I have no preference here and I'm happy to implement what you API guys decide :-)

Andreas Mueller [email protected] schrieb:

I agree. I'm also not sure what it is we actually want in the sparse case.
I think the default option should keep sparsity structure, as this is how we did it in other cases afaik (RandomizedPCA for example), and the user needs to be explicit to destroy sparsity structure.

So maybe we want an with_offset option which is True for dense and False for sparse input.

If the input if strictly positive, between 0 and x and the given range is [0, a] then the right thing will happen.
If the input is between -x and x and the range is [-a, a] also the right thing will happen.

In all other cases, something a bit weird would happen, though
If the input is between -x and x and you ask for [0, a] you would get [0, a/2] I think... well...

—
Reply to this email directly or view it on GitHub.

Any opinion @larsmans ?

I think center="auto" with a default value of issparse(X) and an exception for the combination of True and dense arrays would be fine.

I wouldn't call it 'center'. And you probably mean "sparse" + True is an exception, right?

It's time to get this merged. @amueller, what name do you suggest?

I would have called it with_offset (similar to with_mean).

Hope it's all right now...

Could you please rebase on top of master (or merge master to your branch) and fix any conflict so as to make github happy?

@ogrisel rebase done, travis happy ;-)

Apart from the lack of documentation for the with_offset option, it looks good. I am not sure about the fact that the behavior is not the same for sparse matrix and dense arrays is different. A naive user would likely expect the default behavior to be the same (as long as min_ == 0).

@ogrisel we discussed the with_offset behavior above. Summary: My first implementation was to keep the behavior the same for dense/sparse. The complicating issue here is that sparse matrices include implicit zeros, which needs to be taken into account. You could then argue that sparse matrices are /meant/ to have zeros, so zero should always be part of the input range. A sparse matrix which does not have zeros would then behave differently than its dense counterpart. The original implementation had a parameter assume_contains_zeros, with default True for sparse matrices. That was too complicated to document and non-intuitive for naive users. I think the solution as it is now (by default, only use scaling for sparse) is the better of the two evils.

Still better documentation (with example) is required because right now I am not sure what it actually does.

@temporaer do you plan to finish the doc work on this PR or would you like someone else to take over the PR from there?

@ogrisel when I tried last, it turned out to be more than just doc work. I'll try to to produce something discussable in the next few days.

Thanks!

Hi there!

I'm working on something similar over in #2514, and I think if we might end up replicating effort (I was in fact already looking into merging your PR into mine) :) Maybe we should hash something out together before we both waste time?

Where are you currently stuck/having problems?

Tonights favoured (by me) solution to the semantic problems: if data is sparse and the offset is non-zero, raise a value error. That means that on the off-chance that a sparse array doesn't contain a zero (and it is scaled between, say 0 and 1), it will bail.

Pro:

• when it computes something, it is consistent with the dense case
• It never makes a matrix dense
• If it computes something, the result is very natural

Con:

• it might fail during a GridSearch.

@larsmans @ogrisel wdyt?

The second best option: make MinMaxScaler densify if it has to and make an additional estimator that only scales data.

Ok even better (though deprecation cycle inducing) idea: remove MinMaxScaler as no-one needs that functionality anyway. Instead introduce MaxAbsScaler, that scales the data to have a given maximum absolute value, and never adds anything to the data.

I personally was always wondering why the option to add something to the data existed, I figured someone must've had a use case for it. What I came up with last night was making the behavior dependent on the feature_range argument. On dense input, keep the current behavior, while on sparse input:

• throw execption if range is [+a, +b] or [-a, -b] (i.e., does not cross zero, hence would destroy sparsity)
• throw exception if range is [0, +a], but the data contains values < 0: (i.e., it would destroy zeros in at least one feature).... same if range is [-a, 0] and data contains values > 0.
• if range is [-a, +b], scale all nonzero values (i.e., zeros will be preserved).... this behavior is different from what the dense part does, but it should be obvious why and it is easy to document/grasp for users.

When I first implemented this, I had something quite like Andy's solution. With one addition, however: I had a parameter assume_data_contains_zeros, which was True by default for sparse X. If the empirical data range did not include zeros, it would be enlarged to contain it. I thought it was intuitive since sparse data should be allowed to contain zeros anyway.

Pro:

• more stable than Andys solution (less likely to fail during grid search)

Con:

• behavior on dense and sparse, but otherwise identical X may differ if assume_data_contains_zeros is left to default (False for dense, True for sparse)

Different behavior on sparse and dense has turned out to be very confusing for users. I've been trying to get rid of that everywhere I can.

The MaxAbsScaler idea is interesting. We can keep the MinMaxScaler as it is to not break the backward compat but recommend to use MaxAbsScaler in the doc as it can support dense and sparse data in a consistent way by default.

Does anyone have a usecase that needs the current functionality of MinMaxScaler? I think I coded it this way, but I have no idea why.

Superseded by #4828.