This seems like a strange place to hardcode the datafusion version.

Well, as per the comment above, it's tricky to do better, and doing it that way is not Bad(tm) (aka tests will not let us forget about updating it).

Since this is mostly about the rust version being compatible with python if we update the rust and forget to update the pyproject this check would pass but we'd still segfault right? I wonder if it makes sense for us to have some kind of compatible_datafusion function on the rust side we expose to python that we can call for this check. Whether that returns major versions or full strings tbd.

That's correct. Ideally we'd have some check. In practice, updating datafusion basically is a major deal, with pyarrow, arrow-rs, and lancedb updates needed. It takes Tim like half a day to sort out. So sure, some automation would be great, but would also have little value short of solving the entire thing (aka Good Luck(tm)).

it's checking [1], isn't that a zero here, or what does the split do exactly?

that's the correct thing, I want "47.0.0" here, not "datafusion"

To answer the question, the split occurs at "==", as per the "==" literal passed to .split() :)

ah, I was misreading it. So but that means we're checking the full 47.0.0 string, isn't that way too aggressive?

I mean minor & patch updates should be fine?

In general, yes. But given our pin (which is exactly the same in pyproject) and datafusion version scheme, no.

If we agree to have packaging as additional dependency, we can have a spec compliant version check here instead.

I think this all target was actually broken when I added it and has been silently ignored.

It looks like it should be:
all = ["rerun-sdk[notebook]", "rerun-sdk[datafusion]"] however that then hits the rerun-notebook bootstrapping issue our pixi config calls out so we probably need to remove the [all] in our pixi. If that's too big a detour here I can file a ticket (or just fix it) separately.

I totally agree, this all target is fubar. I tiny bit less so now, but I would much rather remove it entirely tbh. Out of scope here though.

Ok I added it so I can file a ticket post coffee. It seemed like a nice convenience but annoying to test with our pixi setup.

Related: we probably should have the datafusion dep for the notebook extra.

Ya? I don't see where we depend on datafusion for the rerun-notebook package. They seem separate to me. We can probably discuss elsewhere though.

Suggestion:

        expected_df_version = CatalogClientInternal.datafusion_major_version()

        datafusion_version = int(version("datafusion").split(".")[0])
        if datafusion_version != expected_df_version:
            raise RerunIncompatibleDependencyVersionError("datafusion", datafusion_version, expected_df_version)

And then in rerun_py/src/catalog/catalog_client.rs in PyCatalogClientInternal

    #[staticmethod]
    pub fn datafusion_major_version() -> u64 {
        datafusion_ffi::version()
    }

I tested this locally, including verifying it fails with another DF version.

Ah, this is much nicer, yes! I'll make the change.