I am hoping this is not a complex solution, as we store all the scores, times and sizes as separate compact numpy arrays, saving time and space.

But expose the list of dict format that we want users to see as a property or a generator function or via __iter__ itself.

>>> list(gs_instance.search_results_)
[{'candidate_id' : 42, 'all_scores' : np.ndarray([0.4, 0.8, 0.9]), 'mean_score' : 7.0,
  'parameters' : {parameter_dict}},
 ...]

And we can also expose a pandas dataframe for the users to further exploit the search result. (gs_instance.search_results_.get_my_panda())

And provide additional statistics in the form of gs_instance.search_results_.get_that_stat()

I think we should create a dict of arrays that the user can call pd.DataFrame on as I said elsewhere.
@jnothman brought up the fact that if we have lists of grids in the grid search, not all models share parameters. (I think the list of grids is the only case that can happen, right?).

Your list of dict doesn't have that problem. But what is the result of get_my_panda? That would have the same issue.
And how do you compute stats? If you need to iterate over the list each time you want to compute something, that might take a while. I'd rather parse everything into an efficient data structure first.

We could have a place-holder value for parameters that are not active, like "not_active".
That would make this array into a dtype object array, which is not the best. But if the parameter was a string, it was already of that type. If it was an int, well, then we lost something.
If a user wants statistics, they can convert to a pandas dataframe where "not_active" is the "missing" value.

Instead of the placeholder, we might want to use masked arrays, which wouldn't need the dtype conversion for missing values. @rvraghav93 can you check how well masked arrays can be converted to dataframes?

Looks promising: http://pandas.pydata.org/pandas-docs/stable/dsintro.html#from-a-series

So why not a dict of masked arrays? I think actually having masks is a "rare" case, so most people don't have to worry about it too much. And if people actually convert to data frames, it should all be fine.
We need to make sure that pandas doesn't map np.NaN to the same missing type. If it does, we need to be explicit about that in our docs. But in the masked array we produce, they will be distinct.

Thanks heaps for your comments!

Ok if I understand your comment correctly, are you suggesting we have one axis for one parameter and whichever point (combination of parameters) doesn't make sense exist (esp. if its a random search) we mask it to nan?

Indeed this would make gathering statistics much easier.

one key in the dictionary or one column in the dataframe for each parameter. And if the parameter is not set, this is masked (not NaN).

And if the parameter is not set, this is masked (not NaN).

You mean left to the default value?

I'm guessing @amueller means "when a parameter is not a part of the parameters being used in this current iteration", a case that can emerge when using lists-of-dicts.

Otherwise there is no real interface (or any point, really) to leave a parameter to its default value.

So am I right we want something like:

kernel    gamma    degree
=========================
'poly'      -        2
'poly'      -        3
'rbf'     0.1        -
'rbf'     0.2        -
'rbf'     0.3        -

where dashes are masked values and each column is an array? (e.g. {'kernel': ['poly', 'poly', 'rbf', 'rbf', 'rbf'], ...})?

Oh so we are trying to capture the grid/random search process? (This way we can stop early or warm start too?)

i.e. Are you suggesting that we do -

# search progresses this way ---> 
{'param_a' : ['foo', 'bar', 'MASKED'],
 'param_b' : [0, 0, 1],
 'all_scores' : [[0.8, 0.9], [0.9, 0.8], [0.8, 0.8]],
 'aggregated_scores' : [0.85, 0.85, 0.8]
}

In this case why not simply repeat 'bar' in that ndarray? (So as to preserve the information that we are not changing that parameter in the current iteration?)

@vene Ah your edited comment explains it clearly! Thanks!

One question. Why do we need a masked array? Why can't we use nan in those places? As converting to pandas will anyway convert the masked elements to nan.

Could we also have a dict element 'candidate_rank' to rank the parameter settings (candidates)?

No strong opinion but that should be pretty straightforward to do if you have a column that has the mean_validation_score no?

I haven't had a deep thought about any possible disadvantages, but in any case @amueller 's proposal on the dict of masked_arrays seems great.

As for masked arrays, if someone wants to operate on search_results_ directly rather than on the converted dataframe, then...

@MechCoder is right, we absolutely don't want to require pandas, and nans confuse argmin/argmax computation. That, plus the same reasons we ran into in the missing value representation discussion, make nans not a fun choice IMO.

Thanks @MechCoder @vene for the inputs!!