Would you mind elaborating a bit more on what the benefit of may_evaluate is compared to explicitly allowing certain functions such as shape (unless we reuse the semantics from the array-API standard's shape function?) and to_array to optionally materialize?

Sure - anything related to automatic execution is going to result in people accidentally double-computing things

Quick example:

df: DataFrame
features = df.drop_columns('target').to_array()  # in Polars, triggers the whole DAG behind `df`
target = df.col('target').to_array()  # in Polars, also triggers the whole DAG behind `df`

as opposed to

df: DataFrame
df = df.maybe_execute()
features = df.drop_columns('target').to_array()
target = df.col('target').to_array()

In the first example, Polars would push down the 'target' selection

Note that for your library, you'd be free to ignore maybe_execute here, because you can do df.drop_columns('target').to_array() lazily

I'm afraid I still don't quite see it, sorry! How would you implement to_array in your second example to take advantage of the maybe_execute hint assuming that (a) to_array returns an array-API compliant array and (b) that array is a numpy array for the pandas-backed-dataframe implementation in question?

Would there be some cache associated with df after the maybe_execute call? If so, would that not deteriorate the user experience with respect to memory consumption? I.e. the user may expect the memory of the features array to be GC'ed after features went out of scope, but instead it would live for as long as df does.

Thanks for asking, this will help clarify things

So, let's consider the following cases

case 1: lazy dataframe, with lazy array counterpart, which requires computation for bool(col.std())

I think Dask would be an example of this. They have a lazy array, but bool(df['a'].std()) raises telling you to call .compute.

You could have may_execute just set a boolean flag, which is then ignored for Column.to_array and DataFrame.to_array but is respected for Column.std().__bool__()

Something like:

class Scalar:
    def __bool__(self):
        if self.parent_df.may_execute:
            self.parent_df = self.parent_df.compute()
            return bool(value)
        else:
            raise RuntimeError("please call `may_execute` on the parent dataframe first")

class Column:
    def to_array(self):
        # ignore `self.may_execute` here, as `to_array` doesn't need it
        return self.column.to_array()

    def std(self):
        return Scalar(self.column.std(), parent_df = self.parent_df)

case 2: lazy dataframe, with eager array counterpart, which requires computation for bool(col.std())

Could just alias may_execute to the underlying collect / compute call, e.g.

class DataFrame:
    # [...]
    def may_execute(self):
        return DataFrame(self.dataframe.collect())

case 3: eager dataframe, everything's eager

may_execute is a no-op

class DataFrame:
    # [...]
    def maybe_execute(self):
        return DataFrame(self.dataframe)

EDIT

As noted below, the above (case 1) may not be a great idea anyway #307 (comment)

Thanks for your comments

I've excluded to_array from this PR, as it's probably not necessary at this point

Conversely, the if df.col('a').std() > 0: call is really necessary to resolve (#305 (comment))

I think may_execute is too ambiguous here. It just implies that the dataframe may hit a blocking operation at some point, but doesn't clearly indicate that it should explicitly execute and block or otherwise do something to influence downstream usage. Also, does doing another operation after may_execute invalidate the may_execute flag?

If we look at lazy execution frameworks (ignoring Ibis since it defers to different backends for actual implementation / execution), they all have a method like cache (Polars, Spark) / persist (Dask, Spark) that don't block but explicitly instruct that all future usage after that operation will not retrigger computation before that operation.

What would your thoughts be if we added a method like cache that is a no-op for eager DataFrames, which gives a better path to avoid performance pitfalls in lazy dataframes, but then generally allowed operations like to_array and bool(Scalar) to always allow running?

Initial thought - sounds good! Will try it out in my implementation / skrub and see where this takes us

Initial thought - sounds good! Will try it out in my implementation / skrub and see where this takes us

Just tried, and no, polars.LazyFrame.cache doesn't persist the dataframe into memory, see pola-rs/polars#2842

Having said that, maybe may_execute could just defer to .collect in Polars and persist in Dask?

Having said that, as far as I can tell,

df: dask.DataFrame
df = df.persist()

is very roughly equivalent (ok, not exactly, as Dask would write to the cluster rather than to your local machine, but bear with me)

df: polars.LazyFrame
df = df.collect().lazy()

Maybe .persist is OK then, and we document the list of methods which implicitly force computation

So then

# DataFrame Standard
df: DataFrame
df = df.persist()
features = []
for column_name in df.column_names:
    if df.col(column_name).std() > 0:
        features.append(column_name)
return features

would be roughly equivalent to

# polars
df: polars.LazyFrame
df = df.collect().lazy()
features = []
for column_name in df.column_names:
    if df.collect()[column_name].std() > 0:
        features.append(column_name)
return features

and

# Dask
df: dask.DataFrame
df = df.persist()
features = []
for column_name in df.column_names:
    if df[column_name].std().compute() > 0:
        features.append(column_name)
return features

And maybe that's fine?

Yea, this is more or less what I had in mind. There's still a footgun if someone doesn't use the method, but it at least gives a standard compliant way for folks to write code that nicely works across both eager and lazy implementations without introducing any implementation burden onto any of the libraries.

There's still a footgun if someone doesn't use the method

Sure but we could raise if persist hasn't been called earlier?

e.g.

df: DataFrame
features = []
for column_name in df.column_names:
    if df.col(column_name).std() > 0:  # raises, tell you to call `persist` on parent dataframe
        features.append(column_name)
return features

Correct way:

df: DataFrame
df = df.persist()
features = []
for column_name in df.column_names:
    if df.col(column_name).std() > 0:
        features.append(column_name)
return features

I think the real footgun would be calling .persist within a loop:

df: DataFrame
features = []
for column_name in df.column_names:
    if df.persist().col(column_name).std() > 0:
        features.append(column_name)
return features

so this is why the "use .persist as late and as little as possible" rule (which we'd document) should still apply

EDIT: though even this last case could be prevented by erroring if persist is called when is_persisted = True. Footguns are still possible, but whatever, if we documented the best practice and have examples, I think it's OK EDIT2: the previous comment doesn't hold because operations here aren't inplace

Sure but we could raise if persist hasn't been called earlier?

e.g.

df: DataFrame
features = []
for column_name in df.column_names:
    if df.col(column_name).std() > 0:  # raises, tell you to call `persist` on parent dataframe
        features.append(column_name)
return features

What are the rules for propagating a persist status vs invalidating it? I think it would be difficult to have a set of cohesive rules for this.

It doesn't need to be part of the standard, but an implementation could raise if you try to bring a scalar into Python without having called persist at some point between __dataframe_consortium_standard__ and the call which forces computation

If we look at lazy execution frameworks (ignoring Ibis since it defers to different backends for actual implementation / execution), they all have a method like cache (Polars, Spark) / persist (Dask, Spark) that don't block but explicitly instruct that all future usage after that operation will not retrigger computation before that operation.

Trying to catch up here. This seems like a reasonable thing to add, given that all lazy libraries seem to have it. I'm not sure if solves the same problem as may_execute(); there are two more or less orthogonal things here:

1. Avoid recomputing expensive calls more than once in lazy implementations
2. Allow writing code where method calls that cannot be kept lazy by a library do work

It looks to me like (1) is solved by .persist, while for (2) it's not yet 100% clear to me from the discussion above. It will solve the problem if it returns something that, for Polars, is neither a LazyFrame nor an EagerFrame. But rather a lazyframe with a few methods that are able trigger execution. I.e. something like:

class LazyFrame:
    ...
    def persist(self) -> PermissiveLazyFrame:
        return self._to_permissive()

class PermissiveLazyFrame(LazyFrame):
    def __bool__(self) -> bool:
        return self.collect().__bool__()

    def __int__(self) -> int:
        return self.collect().__bool__()

    def __float__(self) -> float:
        return self.collect().__bool__()

    def to_array(self) -> numpy.ndarray:
        return self.collect().numpy()

    # add some magic here to ensure that for all other method calls,
    # the return type is PermissiveLazyFrame, not LazyFrame

It could be a short implementation, the above is pretty much all that's needed. This would also answer be in line with Marco's answer on when things raise (if .persist has been called at least once) - because LazyFrame continues raising. Is my understanding correct there?

For the standard, the description of __bool__ & co would then still be slightly awkward, something like: "returns a boolean scalar; may raise for lazy implementations; is guaranteed not to raise if .persist() has been called on the dataframe before unless the library is not able to execute anything eagerly."

Something like that (but note that it's Scalar.__bool__, not LazyFrame.__bool__)

I've tried this out in dataframe-api-compat anyway - if anyone fancied trying it out (pip install dataframe-api-compat - I'm releasing a bit liberally whilst we have no users) and seeing if it matches their expectations, that'd be really helpful

little demo:

# t.py
from __future__ import annotations

import pandas as pd
import polars as pl

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from dataframe_api.typing import SupportsDataFrameAPI
    from dataframe_api import DataFrame

dfpd = pd.DataFrame({'a': [1, 1, 1], 'b': [4,5,6]})
dfpl = pl.DataFrame({'a': [1, 1, 1], 'b': [4,5,6]})

def this_raises(df_raw: SupportsDataFrameAPI):
    df = df_raw.__dataframe_consortium_standard__(api_version='2023.11-beta')
    features = []
    for column_name in df.column_names:
        if df.col(column_name).std() > 0:
            features.append(column_name)
    return features

def this_runs(df_raw: SupportsDataFrameAPI):
    df = df_raw.__dataframe_consortium_standard__(api_version='2023.11-beta')
    df = df.persist()  # type: ignore
    features = []
    for column_name in df.column_names:
        if df.col(column_name).std() > 0:
            features.append(column_name)
    return features

def this_runs_but_dont_do_it(df_raw: SupportsDataFrameAPI):
    df = df_raw.__dataframe_consortium_standard__(api_version='2023.11-beta')
    features = []
    for column_name in df.column_names:
        if df.persist().col(column_name).std() > 0:  # type: ignore
            features.append(column_name)
    return features

Then (note: tracebacks shortened):

In [1]: this_raises(dfpd)
---------------------------------------------------------------------------
ValueError: Method scalar operation requires you to call `.persist` first on the parent dataframe.

Note: `.persist` forces materialisation in lazy libraries and so should be called as late as possible in your pipeline, and only once per dataframe.

In [2]: this_raises(dfpl)
---------------------------------------------------------------------------
ValueError: Cannot materialise a lazy dataframe, please call `persist` first

In [3]: this_runs(dfpd)
Out[3]: ['b']

In [4]: this_runs(dfpl)
Out[4]: ['b']

In [5]: this_runs_but_dont_do_it(dfpd)
Out[5]: ['b']

In [6]: this_runs_but_dont_do_it(dfpl)
Out[6]: ['b']

Error messages needs sorting out, but this is the idea

I've updated, and removed the "propagation" part. We can talk about that next time - for now let's just get persist in?

I think people agreed on everything in this PR

thanks all, merging then

we can discuss propagation (or lack of) next time, but I'm glad we've been able to agree on this. it's something to be proud of. well done all! 🎉