sklearn.workflow

sklearn.glue

sklearn.scaffolding

sklearn.compose

sklearn.composition

sklearn.flow

my main problem with flow is the existence of tensorflow

sklearn.stack

sklearn.connectors

sklearn.connect

Hm so I think putting pipeline and FeatureUnion in it for the deprecation is a good thing and possibly the least controversial of it. Then there is the questions of whether we want ColumnTransformer and TargetTransformer in it. If it's basically a rename of pipeline I would expect @GaelVaroquaux to argue against ColumnTransformer. I think it would be good to have it there.

I guess TargetTransformer also makes sense. Given the meta-estimator nature it seems to belong here more than into preprocessing.

Oh and as I said in the other thread, I think a slow deprecation would be good, but I would not use 1.0 as a target.

The thing about ColumnTransformer is that it doesn't do any interesting work itself. And I think users will realise that, and become familiar with the concept that they need glue to aid in feature extraction and preprocessing being differentiated by column, rather than a processor itself.

@GaelVaroquaux I think you should consider that people working with columnar data don't consider themselves people working on columnar data, and have probably never heard that term. Having homogeneous data is the special case for most people that I ever talk to, and people (in particular coming from R or weka) are very surprised that estimators don't handle "columnar data". I think our API choices there make a lot of sense, but I feel that casting this kind of preprocessing as feature extraction will seem very alien to them.

@GaelVaroquaux @agramfort do you wanna vote? Should I unleash hell and tweet the issue? ;) And maybe most importantly, do we want to stall #9012 for this (I don't think so)

I like the idea a start small contrib package for 'advanced' pipelines (I have in the mind also the transform X, y with fit that changes sample size), feature unions etc. Given how long we have talked about this I think that can be a good way to move forward without fear of full project overhead.

Well fit that changes sample size exists already in imblearn and I haven't really seen another application of changing y. The ColumnTransformer is ready and waiting only for a module to live in.
I really don't want to teach again without being able to do proper cross-validation on titanic or adult.

I think we must have ColumnTransform in sklearn, not some other package. But yes, perhaps trialling a more conformant Pipeline can happen outside sklearn. I also think that imblearn's Pipeline model is basically good enough and we should adopt it here, but that's sort of off topic.

Imputing missing values on categorical and continuous variables without custom code. And/or not scaling one-hot-encoded variables. And ideally one-hot-encoding variables within the cross-validation, but that's not 100% necessary.

It's possible but you need a custom column selector. Basically most reasonable workflows I can think of need to treat categorical and continuous variables different during preprocessing. How do you do that right now?

With ColumnTransformer I can do something like (untested)

categorical = X.dtype == object
preprocess = make_column_transformer(
    (make_pipeline(Imputer('median'), StandardScaler()), ~categorical),
    (make_pipeline(Imputer('most_frequent'), CategoricalEncoder()), categorical))
model = make_pipeline(preprocess, LogisticRegression())

That's the most simple workflow that I can think of that works with "columnar" data, i.e. most of the things my students ever see.

The other common case that's annoying right now is if training and test data come from different sources (like different csv files), you need to convert to pandas categorical and store the categories (because OneHotEncoder can't deal with strings and CategoricalEncoder requires ColumnTransformer)

If you allow the imputation of categorical to be a continuous relaxation and are fine with doing a StandardScaler on them, you can simplify my pipeline. But that seems not very natural / didactic. Is that what you're doing?

Also see #8540

thanks @amueller for clarifying the use cases. I've never had the use case of train/test data from 2 different CSVs. For teaching on titanic I tend to bypass the pb of column specific scaling as I make them use linear models with float columns and then I make then use trees...

I would put ColumnTransformer / make_column_transformer in preprocessing for now.

It just dawned on me that the thing all these estimators have in common is that they're about transformers, and that they're meta-estimators. sklearn.transform_ensemble would be perfect were it not so verbose.

sklearn.transform

sklearn.combine

the thing all these estimators have in common is that they're about transformers, and that they're meta-estimators

Or you could also phrase it as: they're all estimators composed of different transformers / estimators -> sklearn.compose

But 'combine' is maybe an easier way to say the same as 'compose'

(combine could be confused with ensemble where compose can less so)

I like compose.

@amueller if you vote for that above, then it is leading in the votes :-)

hm I guess on the first read I didn't like it? I didn't remember it being up there lol.

Maybe @ogrisel wants to vote?

I'm willing to do that... but you take the responsibility ;)

Any more comments, or somebody who takes a decision?
This is a blocker for the ColumnTransformer (#9012), unless we decide to for now put it in one of the existing modules and move it later if we decide on this one.

Well i get the impression people like sklearn.compose, conditioned on the assumption that we're doing this.
@GaelVaroquaux remains rightly concerned that users won't know where to find what they need, but I think that's already true for something like FeatureUnion, and search engines / stack overflow help a lot.

How do we define sklearn.compose? Home to metaestimators for composing transformers together and with other estimators, including being the future home of Pipeline and FeatureUnion versions that do not alter their input parents (in the fabled version 1.0)

I'd like to see this go ahead, if only to break an impasse.

I hope that I didn't give the impression that I was opposed to "compose". It's my favorite name.

I am a bit unhappy that I have the impression that we are changing things for cosmetic reasons. That's terrible for long-term users (right now, for a review, I am rewamping code that I wrote 5yrs). It's the error that Python made in the transition from 2 to 3.

But if there is a general idea that pipeline must die, let's call the replacement compose, and move forward.

But if there is a general idea that pipeline must die, let's call the replacement compose, and move forward.

I think it only makes sense if there is agreement that the current Pipeline API should change (cloning its steps) ànd if we think that changing it inplace in pipeline.Pipeline is a too disruptive change.

including being the future home of Pipeline and FeatureUnion versions that do not alter their input parents (in the fabled version 1.0)

I don't think we need to wait with putting Pipeline and FeatureUnion in compose ? If we add this module now, we can add an adapted version of Pipeline/FeatureUnion there directly as well?

Is it worth organising some kind of live meeting to resolve the issues here?

Sorry I'm not monitoring the issue tracker. If you want to meet, I can be reached by email.
It sounds to me like there is agreement and what to do, with the only thing that's left open being:

But issues remain about how to construct a Pipeline from pre-fitted
components

That's the freezing estimators question, right?

should FunctionTransformer go there as well?