Coverage increased (+0.0%) when pulling e5e363d on IssamLaradji:Extreme-Learning-Machines into 68b0a28 on scikit-learn:master.

Hi, I am wondering what verbose extreme learning machines should display. Any ideas ?

Thanks

Travis is acting strange, in that it raises an error for test_multilabel_classification(), although, in my local machine, the test_multilabel_classification() method in test_elm runs correctly with 1000 different seeds. Also, the pull request passed the local test after executing make test on the whole library.

Is there a chance that Travis uses libraries different (or a modified version) from the local for testing ?

This might be worth having a look at https://github.com/dclambert/Python-ELM.

Training squared error loss would seem appropriate for verbose output. Not every estimator has verbose output, though (naive Bayes doesn't because it runs instantly on typical problem sizes).

Coverage increased (+0.07%) when pulling 2be2941 on IssamLaradji:Extreme-Learning-Machines into 68b0a28 on scikit-learn:master.

Thanks, displaying the training error as verbose is such a useful idea.

However, Travis raises an error for test_multilabel_classification(). Is there a chance that Travis uses libraries different (or a modified version) from the local for testing ?

The version of numpy / scipy used by the various travis workers are given in the environment variable of each build. You can see the exact setup in:

• https://github.com/scikit-learn/scikit-learn/blob/master/.travis.yml
• https://github.com/scikit-learn/scikit-learn/tree/master/continuous_integration

@ogrisel thanks I will dig deeper to see where multi-label classification is being affected.

Hi guys, I implemented weighted and regularized ELMs - here are their awesome results on the imbalanced dataset. :) :)

Non-Regularized ELMs (Large C)

Regularized ELMs (Small C)

Pushed a lot of improvements.

1. Added sequential ELM support - with partial_fit
2. Added relevant tests for sequential ELM and weighted ELM

Created two examples.

1. Weighted ELM plot
   

2. Training vs. Testing with respect to hidden neurons
   

Will be leaving the documentation till the end - after I implement the remaining part which is kernel support and after the code is reviewed. Thanks.

@agramfort thanks for your comments. I pushed the updated code.

Updates,

1. ELM is now using ridge-regression as off-the-shelf solver to compute its solutions.
2. Added support for kernels - linear, poly, rbf, sigmoid.
   Is there a way we could reuse the fast, efficient SVM kernel methods?
   Thanks.

There are kernels in sklearn.metrics. The ones in sklearn.svm are buried deep down in the C++ code for LibSVM.

It would be good to have an empirical case where the partial fit actually helps. The incremental fitting is what really makes this code non-trivial. If this is helpful, I'd say merge this with renaming / reassignment of credit, and later refactor into Ridge.

If not, maybe just add a transformer and an example?

For a stateless transformer, I presume the fit is mostly needed for input checking and setting the rng? The rng could be set in the first call to transform, although this might break your common tests. In any case, you can just call fit once outside of the for loop.

So, indeeded, the incremental fitting is useful in the n_features < n_hidden < n_samples regime. But this is the usual out-of-core learning setting: your features are too big so you need to build them on the fly and call partial fit on small batches. @agramfort had an example using polynomial features in his pydata talk :)

If we really want to go the estimator way (rather than the transformer way), there is actually a more elegant and concise way to solve the problem using conjugate gradient with a LinearOperator. This technique can be used to solve the system of linear equations without ever materializing the transformed features of size n_samples x n_hidden. This is because conjugate gradient only needs to compute products between the n_hidden x n_hidden matrix and a vector. This should be like 10 lines of code. See https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/linear_model/ridge.py#L63 for an example of how this works.

Yeah stateless transformers and the common tests don't work well together, they are currently manually excluded, and that is something we / I need to fix.
The problem with setting the random weights on the first transform is that this would break if people want to use a transformer object on two different datasets. n_features is usually inferred in fit and if you use another one in transform that is an error. I guess you could set it in transform unless it was set in fit, and if you explicitly want to use it on another dataset, you have to call fit. That is slightly magic, though.

For better ways to solve the problem: well it could be that n_features is small enough that you could fit n_samples x n_features into ram, but not n_samples x n_hidden. Not sure what typical n_features and n_hidden are.

You are the expert in solving linear problems, I am certainly not, so if there are smarter ways to solve this, then we should go for these.
I didn't mentor this GSoC, I just heard multiple times "this just needs a final review"..

For better ways to solve the problem: well it could be that n_features is small enough that you could fit n_samples x n_features into ram, but not n_samples x n_hidden. Not sure what typical n_features and n_hidden are.

Sorry, when I was talking about generating features on the fly, I was referring to the features generated by the random projection + activation transformer. This is the same setting with polynomial features as well: your original features fit in memory but not the combination features obtained by PolynomialFeatures. But the principle is the same even if you start from your raw data, as long as the transformer used is stateless (e.g., FeatureHasher). In all cases, we loop over small batches of data, transform them and call partial_fit.

For partial fit pipelines, I would only support stateless transformers for the moment.

+1: let's tackle the simple things first.

Did I say I have a plan? ;)
Three possible ways?

1. detect (I have no idea how)
2. annotate as stateless
3. via api: stateless transformers don't need to call fit. Then we don't need to call fit in partial_fit. This would also allow users to provide a statefull transformer fit on a subset of the data.

Can we decide what to do with this PR first?
@IssamLaradji put a lot of work into it, and it has been sitting around for way to long. If we feel that the algorithm isn't suited for a classifier / regressor class, we should see what we can salvage and add transformers / examples etc.

+1 for a transformer on my side. Instead of using a pipeline of two transformers as I initially suggested, we can maybe create just one transformer that does the random projection and applies the activation functions. This should be fairly straightforward to implement. For the examples, showing how to do grid search with a pipeline would be nice. For the name, of the transformer, maybe RandomActivationTransformer?

Do you think the interative ridge regression here has value or are there better ways to partial_fit ridge regression?

The idea of accumulating the n_hidden x n_hidden matrix is nice but this won't scale if n_hidden is large. If we implement a general partial_fit out of this algorithm, this will crash when people try it on high dimensional data like bag of words. We can add it and recommend not to use it when n_features is large. This would still be useful in some settings where n_samples is huge but n_features is reasonably small. For n_features large, I guess one should use SGD's partial_fit.

Ok. So lets do the transformer? @IssamLaradji do you want to do that?
Or do you think you don't have time?

I'm not sure about RandomActivationTransformer. Maybe NonlinearProjection though projection kinda means to a lower dim space. NonlinearRandomFeatures? RandomFeatures?

I don't like RandomFeatures, it's way too generic. From the name I'd expect it to simply ignore X and return random features. Out of all the names here, it seems to me like RandomActivation is the most specific (it best conveys what the object does). (I'd remove the Transformer suffix).

aside @amueller re stateless transformers: for this purpose, transformers
that depend only on the type or number of columns of the input should also
be acceptable, just to make things tricky!

On 9 May 2015 at 01:43, Vlad Niculae [email protected] wrote:

I don't like RandomFeatures, it's way too generic. From the name I'd
expect it to simply ignore X and return random features. Out of all the
names here, it seems to me like RandomActivation is the most specific (it
best conveys what the object does). (I'd remove the Transformer suffix).

—
Reply to this email directly or view it on GitHub
#3306 (comment)
.

@amueller yah sure! I can do the transformer.

So I will open a new pull request for this.

Should the file, containing the algorithm, be under the scikit-learn main directory ?
I mean, would it be something like from sklearn import RandomActivation ?

Would the parameters be something like,
-weight_scale; which sets the range of values for the uniform random sampling.

• activation_function; which could be identity, relu, logistic and so on.

PS: I think for ridge regression there is also feature-wise batch support which scales with n_features rather than n_samples.

One possible place would be the pre-processing module.

Actually how about putting it in the neural_network module?

sounds good.

right, it's only used by neural network algorithms as far as I know, so having it in neural_network module is better imo.

#4703 this is a rough implementation of the RandomActivation algorithm

How about "RandomBasisFunction" as a name?

The decoupled approach was the same approach of https://github.com/dclambert/Python-ELM
You had a "random_layer" and you could also pipeline it before a Ridge regression.

Moreover, it also included a MELM-GRBF implementation.

Just FYI:
Recently Anton Akusok et al. has implemented ELM in python with MAGMA-based acceleration, under the name of hpelm (pypi:https://pypi.python.org/pypi/hpelm).

I don't think this will ever get merged. Closing. Feel free to reopen if you disagree.