ping @amueller

On hindsight, I think something like loss_function which takes in y_true and ypred as input and returns the loss would have been better, since in that way I can use functions in sklearn.metrics directly.

Tests should fail because of this (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/tests/test_ransac.py#L296) . I find it odd that the residual metric for a 1-D array, expects a 2-D array.

I'm not sure what the cleanest way to proceed is, except for deprecating the behavior.

ping @ahojnnes . Would be great to have your inputs.

test?

I can add a test if we decide what to do with the current behavior.

The current behavior for residual_metric accepts a 2-D array and returns a 1-D array even for mono-output y.

no opinion ...

deprecate?

See: #2025

also pinging @jnothman and @arjoly as they reviewed the earlier PR

@MechCoder travis is unhappy

Can you add tests?

@amueller Tests are failing because I do not know what to do with the existing behavior for 1-D array.

The parameter residual_metric accepts a callable that accepts a 2-D array and returns a 1-D array.
This behavior is for both 1-D and 2-D arrays.

For example, now if I need to support arbitrary residual metrics for 1-D array, I need to do this

res_met = lambda dy: np.sum(dy**2, axis=1)
ransac = RANSACRegressor(min_samples=5, residual_metric=res_met)
ransac.fit(X, y)

which is unusual, don't you think?

@agramfort @arjoly IRL @MechCoder and I just discussed the current behavior. We thought it might be more scikit-learn style to pass a scorer instead of the residual_metric. That would allow reuse of the functions in the metrics module.
We could deprecate residual_metric and introduce scoring. That makes writing your own one slightly harder, though.
Alternatively, we could pass a score_func which is just one of the metrics functions. That would mean we can't use strings, though. Wdyt?

@agramfort

# Previous
from sklearn.linear_regression import RANSACRegressor
res_metric = lambda dy: np.mean(np.abs(dy.reshape(-1, 1)))
ransac = RANSACRegressor(residual_metric=res_metric)
ransac,fit(X, y)

# Suggested approach 1
ransac = RANSACRegressor(scoring=mean_absolute_error)
ransac.fit(X, y)

# Suggested approach 2
scorer = make_scorer(mean_absolute_error, greater_is_better=False)
ransac = RANSACRegressor(scoring=scorer)
ransac.fit(X, y)

Then, how do you suggest we provide string inputs?

sounds good to me as well.

I'm happy to use callables that take y_true and y_pred. It's just that I don't think we use this kind of function as an option anywhere else. But I don't have a strong opinion.

@amueller @agramfort I've made changes. Please review.

that's fine with me. I'll let @amueller validate.

@amueller I can haz reviews?

Left you two comments, otherwise LGTM.

Two +1's from @agramfort and @ahojnnes . Will merge when Travis passes