Codestin Search App

MaxwellLZH · 2021-04-18T08:28:08Z

Reference Issues/PRs

This is a fix to issue #19905. Now we check if the input is string or object first using X.dtype.kind in ('U', 'S', 'O'). X and X_roundtrip are converted to object type before equality check because for some reason when array.dtype=='U', np.nan equals np.nan (shown in the following snippet)

a = np.array(['abb', 'b', 'd', np.nan])
b = np.array(['abb', 'b', 'd', np.nan])
print(a.dtype.kind)  # U
a == b   # array([ True,  True,  True,  True])

Corresponding test case is also added.

glemaitre

I think that we can limit ourselves to object dtype only but make sure that it works for a list, an array, and a series (and as an extension on a dataframe).

Please add an entry to the change log at doc/whats_new/v1.0.rst. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:.

glemaitre · 2021-07-23T14:39:26Z

+
+        if not hasattr(X, 'dtype'):
+            X = np.asarray(X)
+        is_object_type = X.dtype.kind in ('U', 'S', 'O')
+        if is_object_type:
+            # need to coerce to Object type before elementwise comparison
+            # because when dtype.kind = 'S', np.nan == np.nan returns True
+            valid = np.alltrue(X.astype('O') == X_round_trip.astype('O'))
+        else:
+            valid = _allclose_dense_sparse(X[idx_selected], X_round_trip)


I think that we could simplify the code:

X = np.asarray(X) if not hasattr(X, "dtype") else X idx_selected = slice(None, None, max(1, X.shape[0] // 100)) X_round_trip = self.inverse_transform(self.transform(X[idx_selected])) if np.issubdtype(X.dtype, np.number): valid = _allclose_dense_sparse(X[idx_selected], X_round_trip) else: valid = np.all(X == X_round_trip)

Co-authored-by: Guillaume Lemaitre <[email protected]>

glemaitre

LGTM

thomasjpfan · 2021-07-28T21:21:50Z

+        if np.issubdtype(X.dtype, np.number):
+            valid = _allclose_dense_sparse(X[idx_selected], X_round_trip)
+        else:
+            valid = np.all(X == X_round_trip)


If inverse_func returns a dataframe of strings & floats, this can fail for floats:

import pandas as pd import numpy as np X = pd.DataFrame({"a": ["first"], "a_int": [1], "a_float": [0.3]}) X_round_trip = pd.DataFrame({"a": ["first"], "a_int": [1], "a_float": [0.1 + 0.1 + 0.1]}) X_np = np.asarray(X) np.all(X_np == X_round_trip) # False

I would prefer not to deal with mixed types when checking for the inverse in FunctionTransformer. I think we can error when X_round_trip is a not a numerical ndarray and suggest check_inverse=False.

WDYT @glemaitre ?

This is probably the best. I would have like to support when we have only strings but it would be difficult to detect. Only the ndarray with "S" or "U" dtype could be easier to handle.

This still needs to be address. I think we should error when X_round_trip is a dataframe that is not all numerical and suggest check_inverse=False.

Checking the inverse with DataFrames with all numerical dtypes works on main, so we need to be careful to continue to support that use case.

I've updated _check_inverse_transform as suggested, now it raises error when input is not all numerical

glemaitre · 2022-01-28T12:24:04Z

Could you solve the conflicts?

thomasjpfan · 2022-01-30T15:06:07Z


    def _check_inverse_transform(self, X):
        """Check that func and inverse_func are the inverse."""
+        X = np.asarray(X) if not hasattr(X, "dtype") else X


This is backward incompatible because transform (which calls self.func) can be designed to take in DataFrame and calling np.asarray(X) would convert the DataFrame into an ndarray.

(I am assuming that validate is False)

thomasjpfan · 2022-01-30T15:07:05Z

+        if np.issubdtype(X.dtype, np.number):
+            valid = _allclose_dense_sparse(X[idx_selected], X_round_trip)
+        else:
+            valid = np.all(X == X_round_trip)


This still needs to be address. I think we should error when X_round_trip is a dataframe that is not all numerical and suggest check_inverse=False.

Checking the inverse with DataFrames with all numerical dtypes works on main, so we need to be careful to continue to support that use case.

glemaitre · 2022-02-14T10:36:46Z

The documentation CI is still failing. Could you solve this before I take a new look at it?

glemaitre · 2022-02-14T10:37:46Z

  `n_features_in_` and will be removed in 1.2. :pr:`20240` by
  :user:`Jérémie du Boisberranger <jeremiedbb>`.

+- |Fix| :meth:`preprocessing.FunctionTransformer._check_inverse_transform`


You need to move this in 1.1 and check the warning raised by the documentation CI

thomasjpfan · 2022-02-15T01:24:19Z

+        X = np.asarray(X) if not hasattr(X, "dtype") else X
+        if np.issubdtype(X.dtype, np.number):
+            valid = _allclose_dense_sparse(X[idx_selected], X_round_trip)
+        else:
+            raise ValueError(
+                "'check_inverse' is only supported when all the elements in `X` is"
+                " numerical."
+            )
+        if not valid:


I think we can early exit here:

if not np.issubdtype(X.dtype, np.number): raise ValueError( "'check_inverse' is only supported when all the elements in `X` is" " numerical." ) if not _allclose_dense_sparse(X[idx_selected], X_round_trip): warnings.warn( "The provided functions are not strictly" " inverse of each other. If you are sure you" " want to proceed regardless, set" " 'check_inverse=False'.", UserWarning, )

Also we do not need to cast since this function already assumes that X is an array or sparse. (There are calls to X.shape[0] and at the beginning of _check_inverse_transform.

thomasjpfan

Minor nit about the changelog otherwise LGTM

Co-authored-by: Thomas J. Fan <[email protected]>

…it-learn#19916) Co-authored-by: Guillaume Lemaitre <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]>

FIX FunctionTransformer roundtrip check for string data

d1137ac

github-actions Bot added the module:preprocessing label Apr 18, 2021

glemaitre assigned glemaitre and unassigned glemaitre Jul 23, 2021

glemaitre reviewed Jul 23, 2021

View reviewed changes

MaxwellLZH and others added 4 commits July 27, 2021 21:52

Update sklearn/preprocessing/tests/test_function_transformer.py

79a3f7b

Co-authored-by: Guillaume Lemaitre <[email protected]>

Merge remote main into this branch

25c2277

simplify code in _check_inverse_transform

1c50692

update whats_new

a69870a

glemaitre approved these changes Jul 27, 2021

View reviewed changes

thomasjpfan reviewed Jul 28, 2021

View reviewed changes

merge main

ba97f7e

github-actions Bot added the cython label Jan 28, 2022

MaxwellLZH added 7 commits January 29, 2022 10:33

solve merge conflict

90a2123

solve merge conflicts

84017db

solve merge conflicts

afbc8be

solve merge conflicts

a5496f9

solve merge conflicts

83785d5

solve merge conflicts

48dc4a0

solve merge conflicts

375bbd2

thomasjpfan reviewed Jan 30, 2022

View reviewed changes

raise error when X has mixed dtypes

e98bb5b

glemaitre reviewed Feb 14, 2022

View reviewed changes

thomasjpfan reviewed Feb 15, 2022

View reviewed changes

MaxwellLZH added 3 commits February 17, 2022 20:09

early exit in _check_inverse_transform

3b0f3ed

update whatsnew

30bb793

Merge branch 'main' into fix/function-transformer-string-input

51d169a

thomasjpfan approved these changes Feb 17, 2022

View reviewed changes

Comment thread doc/whats_new/v1.1.rst Outdated

update changelog for better clarity

8c4f751

Co-authored-by: Thomas J. Fan <[email protected]>

thomasjpfan merged commit dbcd4d5 into scikit-learn:main Feb 17, 2022

glemaitre mentioned this pull request Dec 30, 2022

'DataFrame' object has no attribute 'dtype' #25261

Closed

Uh oh!

Uh oh!

Conversation

MaxwellLZH commented Apr 18, 2021

Reference Issues/PRs

Uh oh!

glemaitre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glemaitre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Jan 28, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glemaitre commented Feb 14, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thomasjpfan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants