Release 0.23.4: Fix migration of old 0.22 rrd files #10350

emilk · 2025-06-24T20:49:19Z

What

NOTE! This is a 0.23.4 release branch, as I want to get these fixes out ASAP.

Problem 1: sorbet meta key name confusion

At some point we renamed the keys of our meta-data, and forgot to update our code:

rerun.archetype vs rerun.archetype_name
rerun.archetype_field vs rerun.archetype_field_name

Problem 2: not writing new meta keys on migrate

Our migration code erroneously would keep the old metadata of all our fields.

The new code in this PR not only fixes this, but also retains all non-rerun metadata, because it's the right thing to do.

TODO

Test that migrated files work on main
Port the second commit to main
Port changelog to main

github-actions · 2025-06-24T20:50:22Z

Web viewer built successfully. If applicable, you should also test it:

I have tested the web viewer

Result	Commit	Link	Manifest
✅	`f18365d`	https://rerun.io/viewer/pr/10350	`+nightly` `+main`

^{Note: This comment is updated whenever you push a commit.}

grtlr

Ah nice, how subtle!

crates/store/re_sorbet/src/sorbet_batch.rs

grtlr · 2025-06-25T10:47:13Z

crates/store/re_sorbet/src/sorbet_batch.rs

+            debug_assert_eq!(new_field.data_type(), column.data_type());
+
+            let mut metadata = old_field.metadata().clone();
+            metadata.extend(new_field.metadata().clone()); // overwrite old with new


Would that work in a setting where we change keys as part of our migration? Because that currently happens on main.

It would, if we would call rewire_tagged_components at the top of SorbetBatch::try_from_record_batch… unfortunately we call it only in ChunkBatch::try_from, which I think is a separate bug (we should always rewire tagged components)

I'll make sure this is correct when I cherry-pick the commit into main

jprochazk

@emilk
This PR needs to be based off of release-0.23.4-alpha.2 😬 it contains some important changes to the build process, but it was never fully released as 0.23.4

I think those changes can also just be cherry-picked in case you don't want to rebase

https://github.com/rerun-io/rerun/tree/release-0.23.4-alpha.2

emilk · 2025-06-25T11:43:43Z

Good catch - I'll rebase

github-actions · 2025-06-25T11:45:45Z

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	`f18365d`	https://landing-k3rfnpmqj-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

emilk · 2025-06-25T11:47:39Z

RC job: https://github.com/rerun-io/rerun/actions/runs/15875469356

addressed

emilk · 2025-06-25T12:22:13Z

Release job failed:

our docker container only contains `pip3`

emilk · 2025-06-25T13:02:08Z

ci_docker/Dockerfile

+    arrow-devel \
+    make automake gcc gcc-c++ kernel-devel \
    cmake \
    curl \
    git \
    git-lfs \
-    libatk-bridge2.0 \
-    libfontconfig1-dev \
-    libfreetype6-dev \
-    libglib2.0-dev \
-    libgtk-3-dev \
-    libssl-dev \
-    libxcb-render0-dev \
-    libxcb-shape0-dev \
-    libxcb-xfixes0-dev \
-    libxkbcommon-dev \
-    lsb-release \
+    atk at-spi2-atk \
+    fontconfig-devel \
+    freetype-devel \
+    glib2-devel \
+    gtk3-devel \
+    openssl-devel \
+    xcb-util-renderutil-devel \
+    xcb-util-devel \
+    xcb-util-wm-devel \
+    libxkbcommon-devel \
    python3-pip \
-    software-properties-common \
    sudo; \


nit: it would be easier to merge changes to this list if it was sorted

Also a bit weird that some things are grouped (atk, make/automake) and others not (xcb) 🤔

rerun-bot · 2025-06-25T14:37:14Z

Version 0.23.4-rc.2 published successfully.

artifact	install
web app
wheels	pip install rerun-sdk==0.23.4-rc.2
crates	cargo install [email protected] --locked
npm	npm install @rerun-io/[email protected]
docs
py docs
rs docs
cpp_sdk zip

rerun-bot · 2025-06-26T14:37:05Z

Version 0.23.4 published successfully.

artifact	install
web app
wheels	pip install rerun-sdk==0.23.4
crates	cargo install [email protected] --locked
npm	npm install @rerun-io/[email protected]
docs
py docs
rs docs
cpp_sdk zip

### Related * #10350

) ### Related * Part of #10273 * Cherry-picked from #10350 ### What Ensure that all the Rerun-specific metadata that is added to a record batch actually makes it into the record batch * [x] full-check

jprochazk and others added 7 commits June 4, 2025 10:24

downgrade to manylinux2_28

bd10f27

update builds to use new manylinux docker image

ede3e5f

set libc to 2.28

7afecb2

Bump versions to 0.23.4-alpha.1

65767be

put everything on the manylinux container

95eab48

disable pixi caching

bcb4cdf

Bump versions to 0.23.4-alpha.2

31f4d09

emilk added the do-not-merge Do not merge this PR label Jun 24, 2025

grtlr self-requested a review June 25, 2025 10:25

emilk marked this pull request as ready for review June 25, 2025 10:32

emilk changed the title ~~Fix migration of old 0.22 rrd files~~ Release 0.23.4: Fix migration of old 0.22 rrd files Jun 25, 2025

grtlr approved these changes Jun 25, 2025

View reviewed changes

crates/store/re_sorbet/src/sorbet_batch.rs Outdated Show resolved Hide resolved

grtlr reviewed Jun 25, 2025

View reviewed changes

emilk force-pushed the release-0.23.4 branch from 56260e9 to fe401eb Compare June 25, 2025 11:16

jprochazk previously requested changes Jun 25, 2025

View reviewed changes

emilk added 3 commits June 25, 2025 13:45

Fix migration of old 0.22 rrd files

e2cfc46

Ensure arrow RecordBatch in SorbetBatch has all the correct metadata

e7b85c5

Add changelog entry

4604a21

emilk force-pushed the release-0.23.4 branch from 099ca13 to 4604a21 Compare June 25, 2025 11:45

Bump versions to 0.23.4-rc.1

a9e4dd5

jprochazk added 4 commits June 25, 2025 14:42

merge dockerfile

8de2f1e

add pip dependency

8d8240c

bump version to 0.16

b721246

use new docker image

013e2c6

jprochazk and others added 4 commits June 25, 2025 14:42

update script

b9a3870

fix syntax error

d863316

use pip3 instead of pip

4a3289d

our docker container only contains `pip3`

Fix nightly build. maybe (#10172)

60939b5

emilk commented Jun 25, 2025

View reviewed changes

Bump versions to 0.23.4-rc.2

88fa639

emilk added the ⛴ release Related to shipping or publishing label Jun 25, 2025

emilk mentioned this pull request Jun 26, 2025

Add test that the right batch metadata is added/removed/retained #10381

Merged

1 task

Bump versions to 0.23.4

b411df2

Bump versions to 0.24.0-alpha.1+dev

f18365d

emilk mentioned this pull request Jun 26, 2025

Add 0.23.4 to changelog #10393

Merged

emilk closed this Jun 26, 2025

emilk added a commit that referenced this pull request Jun 26, 2025

Add 0.23.4 to changelog (#10393)

6564feb

### Related * #10350

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Release 0.23.4: Fix migration of old 0.22 rrd files #10350

Release 0.23.4: Fix migration of old 0.22 rrd files #10350

Uh oh!

emilk commented Jun 24, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 24, 2025 •

edited

Loading

Uh oh!

grtlr left a comment

Uh oh!

Uh oh!

grtlr Jun 25, 2025

Uh oh!

emilk Jun 25, 2025

Uh oh!

emilk Jun 25, 2025

Uh oh!

jprochazk left a comment •

edited

Loading

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025 •

edited

Loading

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

emilk Jun 25, 2025

Uh oh!

emilk Jun 25, 2025

Uh oh!

rerun-bot commented Jun 25, 2025

Uh oh!

rerun-bot commented Jun 26, 2025

Uh oh!

Uh oh!

Release 0.23.4: Fix migration of old 0.22 rrd files #10350

Release 0.23.4: Fix migration of old 0.22 rrd files #10350

Uh oh!

Conversation

emilk commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related

What

Problem 1: sorbet meta key name confusion

Problem 2: not writing new meta keys on migrate

TODO

Uh oh!

github-actions bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grtlr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grtlr Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

emilk Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

emilk Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

jprochazk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

github-actions bot commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

emilk commented Jun 25, 2025

Uh oh!

emilk Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

emilk Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

rerun-bot commented Jun 25, 2025

Uh oh!

rerun-bot commented Jun 26, 2025

Uh oh!

Uh oh!

emilk commented Jun 24, 2025 •

edited

Loading

github-actions bot commented Jun 24, 2025 •

edited

Loading

jprochazk left a comment •

edited

Loading

github-actions bot commented Jun 25, 2025 •

edited

Loading