Thank you for opening this issue and a great summary.

Sorry I misunderstood your approach of using the estimator instance to search in the parameter tree.
It is indeed at least as general as the one I mentioned.

Attaching the parameters to the estimator gets partially rid of the "parallel dictionaries" that you lamented elsewhere.

I took the liberty to enumerate you questions.

1. I would say yes.
2. Uh oh. This is tricky. Some might try to nest one GridSearchCV inside another. Do we want to support this?

For the curious, this actually runs:

from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC
from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.datasets import load_iris

iris = load_iris()

grid1 = {"C": [0.01, 0.1, 1, 100]}
grid2 = {"selectkbest__k": [1, 2, 3]}
because_I_can = GridSearchCV(make_pipeline(SelectKBest(), GridSearchCV(LinearSVC(), grid1, verbose=10)),
                             grid2, verbose=10)

because_I_can.fit(iris.data, iris.target)

1. Do you mean a LassoCV inside a GridSearchCV? Or just running LassoCV. Ideally, I'd say both, but the current interface for defining the parameters in the LassoCV is not a list :-/

a) I'd say yes. Trying out both is a bit awkward then :-/
b) you could bend the syntax as passing a list of dictionaries? I thought about passing a dict but as **kwargs is constructed anyhow, it seems a bit unnecessary.

Not entirely related, but something that would be really cool is to create a dask-graph out of the pipeline and parameters. If we had an interface that would support this....

I think we should not rule out needing to support nested grid search. Trying to think of cases, I could imagine OvR or similar having a grid search within it so that hyperparams are not tied across labels in a multilabel problem, but potentially a grid search outside of that too...?

I'm not sure how to proceed on this.

I've been speaking to the auto-sklearn folks. They really want nested grids. I'm not saying we should to that now, but we should keep it in mind.
Oh wait, you meant something else by nested grids, you meant the thing I wrote above. Never mind, this was irrelevant.

isn't this discussion getting into ideas like this? http://optunity.readthedocs.io/en/latest/notebooks/notebooks/sklearn-automated-classification.html where you have nested search spaces

search = {'algorithm': {'k-nn': {'n_neighbors': [1, 5]},
                        'SVM': {'kernel': {'linear': {'C': [0, 2]},
                                           'rbf': {'gamma': [0, 1], 'C': [0, 10]},
                                           'poly': {'degree': [2, 5], 'C': [0, 50], 'coef0': [0, 1]}
                                           }
                                },
                        'naive-bayes': None,
                        'random-forest': {'n_estimators': [10, 30],
                                          'max_features': [5, 20]}
                        }
         }

Not really the same issue, no. Not sure what you're saying about such structures either.

I just realized that RandomizedSearchCV only accepts a dictionary as input to param_dists making it impossible to have conditions in the grid (e.g. if kernel is rbf explore also parameter gamma with a given distribution). The similar setup is possible in GridSearch by passing a list of dictionaries to the parameters argument. I was wondering if there is an actual technical reason for the symmetry to be lost.

It would make sense for it to be uniformly sampled, the same as what happens when you input a list as value of a parameter key the RandomizedSearchCV parameters argument.