I think this would be more useful and scikit-learn like with a choice of activation function. It's still easy to throw a Ridge on the end, but not quite as fragmented.

Also, I think this PR would be well illustrated by translating examples from #3306 into this model.

yah that's true! will update it

Btw, how about "RandomBasisFunction" as a name?

yah RandomBasisFunction sounds better and more technically appealing

I'd add support for different initialization/randomization algorithms. E.g. The Nguyen-Widrow algorithm would be a nice addition.

have you checked out the travis error? And porting the examples and doc over?

yes, working on it - will push everything all at once. :)

Unfortunately, I caught the flu and had to stay in bed for the last 3 days :(, came at the worst time.

I'm sorry! get well soon! [I've had some virus for the last 2 month, hurray ^^]

Thanks!! that's nice of you ! :)

back on my feet! :)

Will push a semi-complete version of this by tomorrow. I will basically use pipelining with Ridge to create examples very similar to those I had for extreme learning machines.

Reproduced the results showing the affect of varying weight_scale and alpha.

Higher alpha in Ridge, more linear the decision graph appears.
Higher weight_scale in the random activation class, more non-linear the decision graph appears.

Added the varying hyperparameters example, doc, and sanity checks in code.

1. What would be a good test case for random activation ?
2. is there another interesting example that I could add ?
3. is it necessary to have a .rst document dedicated to this, or is the code doc sufficient ?

cheers.

2. what about an example when varying n_activated_features?

Recently, Liu, Gao, etal. (2012) showed that the VC dimension
of an ELM is equal to its number of hidden neurons with probability
one. Thus we are provided a convenient way to compute the
generalization bound of ELM.

Source: http://www3.ntu.edu.sg/home/egbhuang/pdf/ELM-Suvey-Huang-Gao.pdf

Here is the plot, scores in terms of training and testing. The dataset was randomly generated where 100 samples are for training, 50 samples for testing, and 50 is the number of features.
When the number of hidden neurons equals the number of training samples, the training score is 1.0, but the testing score plummets significantly, which makes sense.

 from base import ACTIVATIONS
ImportError: No module named 'base'

I wonder why this is a travis error for python 3.

For python 2.6, this works fine.

Implicit relative imports are no longer available in python 3. See for instance http://stackoverflow.com/questions/12172791/changes-in-import-statement-python3 .

I never use relative imports but I think from .base import ACTIVATIONS would work.

@sveitser awesome! this fixed it!

Why is there an "orphan" softmax function? It's not documented and it's not inside the ACTIVATIONS dict.

@ekerazha I assume it has to do with the last layer of the random neural network algorithm... This PR is the first stage for the pipeline of a random neural network algorithm. I haven't really read that paper but I assume the orphaned softmax will join the party later on ;)

There should definitely be an rst with a narrative documentation. It doesn't matter so much if it is it's own file, there will be a supervised file when merging the mlp anyhow.

Hi @IssamLaradji first of all great work on MLP. That was one of the first algorithms I used here. I just wanted to know if you are still working on this ? Thanks.

Hi @maniteja123, thanks a lot!

Unfortunately, I am in the crunch time season of my PhD :( - therefore it might be a while before I can continue working on this.

You are welcome to work on this if interested :)

Hi @IssamLaradji thanks for the response. I actually am interested to work on this and thanks for the permission to work. All the best for your PhD thesis :)

I found this new "Extreme Learning Machines" implementation https://github.com/maestrotf/pyExtremeLM which is pretty nice.

Is there still interest in including this?

I think this is out of scope for scikit-learn these days.