MAINT use ArgKmin in OutputCodeClassifier - #25196
Conversation
adrinjalali
left a comment
There was a problem hiding this comment.
I agree in principle. I wonder if we can manufacture a test for this.
Apparently, we don't have some test failing ;). Interesting to know that the I will try to see if we can add some better tests. |
| # ArgKmin only accept C-contiguous array. The aggregated predictions need to be | ||
| # transposed. We therefore create a F-contiguous array to avoid a copy and have | ||
| # a C-contiguous array after the transpose operation. | ||
| Y = np.array([_predict_binary(e, X) for e in self.estimators_], order="F").T |
There was a problem hiding this comment.
May you run a benchmark to compare peak memory and runtime performance between main and this PR?
There was a problem hiding this comment.
I don't see any regression in terms of memory usage.
This PR:
peak memory: 364.31 MiB, increment: 103.70 MiB
In 1.2.0:
peak memory: 387.33 MiB, increment: 115.27 MiB
Details
from sklearn.datasets import make_classification
from sklearn.multiclass import OutputCodeClassifier
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(
n_samples=100_000,
n_features=100,
n_informative=20,
n_classes=20,
n_clusters_per_class=1,
random_state=0,
)
ecoc = OutputCodeClassifier(
estimator=DecisionTreeClassifier(max_depth=2, random_state=0),
code_size=3,
n_jobs=-1,
random_state=0,
)
ecoc.fit(X, y)
# %%
%load_ext memory_profiler
# %%
%memit ecoc.predict(X)|
For the case of |
| check_is_fitted(self) | ||
| Y = np.array([_predict_binary(e, X) for e in self.estimators_]).T | ||
| pred = euclidean_distances(Y, self.code_book_).argmin(axis=1) | ||
| # ArgKmin only accept C-contiguous array. The aggregated predictions need to be |
There was a problem hiding this comment.
| # ArgKmin only accept C-contiguous array. The aggregated predictions need to be | |
| # ArgKmin only accepts C-contiguous array. The aggregated predictions need to be |
…_reduction_in_ecoc
…_reduction_in_ecoc
adrinjalali
left a comment
There was a problem hiding this comment.
LGTM, but I think this deserves a FIX changelog entry.
|
@adrinjalali this is quite transparent to the end user. This is a not a FIX but rather an enhancement. |
|
@glemaitre if that were true, this script should never import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics.pairwise import pairwise_distances_argmin
from sklearn.multiclass import OutputCodeClassifier, _predict_binary
def new_predict(self, X):
# ArgKmin only accepts C-contiguous array. The aggregated predictions need to be
# transposed. We therefore create a F-contiguous array to avoid a copy and have
# a C-contiguous array after the transpose operation.
Y = np.array(
[_predict_binary(e, X) for e in self.estimators_],
order="F",
dtype=np.float64,
).T
pred = pairwise_distances_argmin(Y, self.code_book_, metric="euclidean")
return self.classes_[pred]
seed = 0
while True:
seed += 1
X, y = make_classification(
n_samples=300,
n_features=10,
n_informative=4,
n_redundant=0,
random_state=seed,
shuffle=False,
n_classes=4,
)
clf = OutputCodeClassifier(
estimator=RandomForestClassifier(random_state=seed),
random_state=seed,
n_jobs=-1,
).fit(X, y)
print("Seed", seed)
data = X + np.random.normal(size=X.shape)
old = clf.predict(data)
new = new_predict(clf, data)
if not np.all(old == new):
# print the data which has resulted in a mismatch
mask = old != new
print(mask)
print("data", data[mask])
print("old", old[mask])
print("new", new[mask])
breakThis breaks in less than 100 iterations in my runs. |
|
OK, I wanted to be sure but basically it comes from the tie-breaking: Seed 1
Seed 2
Seed 3
Seed 4
Seed 5
[False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False True False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False
False False False False False False False False False False False False]
data [[-1.30339834 2.68637221 3.80043205 -0.86839261 -2.06343496 -3.59474715
0.74143858 -2.98092828 2.93166608 -2.01373733]]
old [2]
old distance [[1.17098249 1.221147 1.00558441 1.00558441]]
new [3]
new distance [[1.00558441]]In I, therefore, agree that this behaviour needs to be documented. Thanks, @adrinjalali for the counter-example. I will add an entry in the change models. |
Make the code using
ArgKmininstead ofeuclidean_distance. The reason will be linked to #25148 where I suspect that we should implement an L1 distance (city-block) as stated in the original paper.As a first step, we can use the
ArgKminimplementation that provides to switch distance and it is optimum since we apply a reduction.