feat: provide inline_transaction model for IO optimizing #4774

majin1102 · 2025-09-19T11:43:13Z

Hi, @wjones127, @jackye1995 , I'm working on issue #4308 #3487

In this PR I wanna proposing the inline_transaction model for IO optimizing. The motivation includes:

Get transaction summary without transaction file IO
Optimize commit by reducing transaction file IO

codecov-commenter · 2025-09-19T12:22:08Z

Codecov Report

❌ Patch coverage is 82.62712% with 41 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.94%. Comparing base (20fceb1) to head (23cdaf7).

Files with missing lines	Patch %	Lines
rust/lance/src/dataset.rs	89.33%	12 Missing and 4 partials ⚠️
rust/lance-table/src/io/commit.rs	70.96%	1 Missing and 8 partials ⚠️
rust/lance/src/io/commit.rs	46.15%	5 Missing and 2 partials ⚠️
rust/lance-table/src/format/transaction.rs	33.33%	6 Missing ⚠️
rust/lance-table/src/io/manifest.rs	80.00%	0 Missing and 2 partials ⚠️
...ust/lance-table/src/io/commit/external_manifest.rs	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4774      +/-   ##
==========================================
+ Coverage   80.84%   80.94%   +0.09%     
==========================================
  Files         330      331       +1     
  Lines      130545   130751     +206     
  Branches   130545   130751     +206     
==========================================
+ Hits       105540   105836     +296     
+ Misses      21269    21165     -104     
- Partials     3736     3750      +14

Flag	Coverage Δ
unittests	`80.94% <82.62%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jackye1995 · 2025-09-19T15:54:36Z

protos/table.proto

+    // if < 200KB, store the transaction content inline
+    bytes transaction_content = 19;
+    // if >= 200KB, store the transaction content at the specified offset
+    TransactionSection transaction_section = 20;


I was thinking we should just keep it as a separated file which seems to be more backwards compatible. Is there any other benefit in storing it at offset of the manifest?

For backwards compatibility we should write both for a while. We can stop writing the external file after adding a feature flag 6+ months in the future. I've described the sequence of steps in #3487

wjones127

I think putting inline in the protobuf is necessary. We can put it at an offset and use existing optimizations to read it without an additional IOP.

wjones127 · 2025-09-19T17:06:21Z

protos/table.proto

+  // The transaction that created this version.
+  oneof inline_transaction {
+    // if < 200KB, store the transaction content inline
+    bytes transaction_content = 19;
+    // if >= 200KB, store the transaction content at the specified offset
+    TransactionSection transaction_section = 20;
+  }


We should just unconditionally store the transaction at an offset, IMO. This makes this simpler, and it's still possible to read the transaction when it is small.

We currently store the index metadata at an offset, and here's how it works right now:

When we read the manifest file, we always read the first block_size bytes (4kb for local fs, 64kb for object storage)

We decode as much as we can from that last chunk. If the entire manifest file is < 64kb, this often means we get the entire content (including the index metadata) in 1 IOP.

If that last chunk wasn't sufficient, we read the remainder.

This basically gives the same effect as the optionally inline message: If it's small, we automatically read the index and transaction information in the first IO request. But if it's large, we do it in a separate request.

Here's the place where we opportunistically read the index metadata from manifests:

lance/rust/lance/src/dataset.rs

Lines 511 to 536 in a385965

// If indices were also the last block, we can take the opportunity to

// decode them now and cache them.

if let Some(index_offset) = manifest.index_section {

if manifest_size - index_offset <= last_block.len() {

let offset_in_block = last_block.len() - (manifest_size - index_offset);

let message_len =

LittleEndian::read_u32(&last_block[offset_in_block..offset_in_block + 4])

as usize;

let message_data =

&last_block[offset_in_block + 4..offset_in_block + 4 + message_len];

let section = lance_table::format::pb::IndexSection::decode(message_data)?;

let indices: Vec<Index> = section

.indices

.into_iter()

.map(Index::try_from)

.collect::<Result<Vec<_>>>()?;

let ds_index_cache = session.index_cache.for_dataset(uri);

let metadata_key = crate::session::index_caches::IndexMetadataKey {

version: manifest_location.version,

};

ds_index_cache

.insert_with_key(&metadata_key, Arc::new(indices))

.await;

}

}

wjones127 · 2025-09-19T17:12:19Z

protos/table.proto

+    // if < 200KB, store the transaction content inline
+    bytes transaction_content = 19;
+    // if >= 200KB, store the transaction content at the specified offset
+    TransactionSection transaction_section = 20;


For backwards compatibility we should write both for a while. We can stop writing the external file after adding a feature flag 6+ months in the future. I've described the sequence of steps in #3487

majin1102 · 2025-09-21T17:07:54Z

protos/table.proto

+  message TransactionSection {
+    // The summary of the transaction including operation type and some kvs in transaction_properties
+    // This is used to get some information about the transaction without loading the whole transaction content
+    map<string, string> summary = 1;


Hi, @wjones127 , What do you think of this part.

Basically it's redundent content for operation and transaction_properties. The case I want to solve now may encount listing historical manifest and transaction summaries. I'm not sure the opportunistically read could actually help this listing case if the manifest is large(actually the larger manifest we might need to accelerate it more).

On the other hand, some information may be useful if we could directly read from manifest like commit message. So I added this summary thing for directly reading from manifest without reading transaction part.
cc @jackye1995

For the list transactions use case, I can’t help like feeling the best thing would be to cache compacted summary data in some secondary file. Like we could generate some Lance file next to the manifest that contains all transaction summaries before it. Then to get most of the history you just need to query that. Then read the transactions of the next few uncached versions. What do you think of that?

How much performance do we want for this use case? I always thought reading information like transaction summary is more for non performance critical workloads, like displaying info to the user, triggerring some optimization jobs based on the action performed, etc. that can stand a few more IOPS.

Like we could generate some Lance file next to the manifest that contains all transaction summaries before it

I had the same ideas. I thought lance might provide a lazy summary file includes tags and branches, summaries and properties. And could use the index framework to maintain this file. This was only an early idea. I could raise a discussion for it. Then we should ignore the summary here.

How much performance do we want for this use case? I always thought reading information like transaction summary is more for non performance critical workloads, like displaying info to the user, triggerring some optimization jobs based on the action performed, etc. that can stand a few more IOPS

For my case, one page displays at least 20 versions. each version should get a summary info includes manifest and transaction. If files seperated, the IO costs might have a magnification of 20x. Let's say read a transaction file costs 100ms，that means 2 seconds magnification at least.

Yeah, I'm inclined to not have the summary there. Just keep it in the transaction. Pulling it out into the manifest means that listing is still O(num versions) IO requests. To make that fast, better to create some mechanism to query it.

non performance critical workloads, like displaying info to the user

I'd argue that displaying info to the user is somewhat performance critical, in that we'd like it to return fast enough to feel responsive.

If files seperated, the IO costs might have a magnification of 20x. Let's say read a transaction file costs 100ms，that means 2 seconds magnification at least.
I'd argue that displaying info to the user is somewhat performance critical, in that we'd like it to return fast enough to feel responsive.

I was thinking we will just make the requests in parallel, so the latency won't really be 2 seconds, more like 200-300ms if anything got throttled and retried. But agree we need to make sure it is fast enough to feel responsive.

I thought lance might provide a lazy summary file includes tags and branches, summaries and properties. And could use the index framework to maintain this file.

that sounds like a nice idea, I originally created the concept of "system index" basically for such use cases.

majin1102 · 2025-09-30T11:54:07Z

Ready for review @wjones127 cc @jackye1995

majin1102 marked this pull request as draft September 19, 2025 11:43

github-actions bot added the enhancement New feature or request label Sep 19, 2025

wjones127 self-assigned this Sep 19, 2025

jackye1995 reviewed Sep 19, 2025

View reviewed changes

wjones127 reviewed Sep 19, 2025

View reviewed changes

majin1102 commented Sep 21, 2025

View reviewed changes

majin1102 force-pushed the inline_transaction_model branch from e7024b1 to fc35d19 Compare September 24, 2025 07:11

feat: support inline transaction

06e1a43

majin1102 force-pushed the inline_transaction_model branch from 9cd17b7 to 06e1a43 Compare September 29, 2025 14:40

Merge branch 'main' into inline_transaction_model

641cb43

majin1102 marked this pull request as ready for review September 29, 2025 15:08

improve some code

e61260c

majin1102 force-pushed the inline_transaction_model branch from a93622b to e61260c Compare September 29, 2025 15:09

This comment was marked as outdated.

Sign in to view

majin1102 marked this pull request as draft September 29, 2025 16:04

majin1102 added 3 commits September 30, 2025 00:36

fix rust ci issue

0aa3cbb

fix tracing ci issue

5c7a75b

fix java ci issue

b4f1005

github-actions bot added the java label Sep 30, 2025

majin1102 added 4 commits September 30, 2025 17:50

spotless:apply

026c038

Merge branch 'main' into inline_transaction_model

5eb532f

fmt --all

45f8562

fmt --all

23cdaf7

majin1102 marked this pull request as ready for review September 30, 2025 11:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: provide inline_transaction model for IO optimizing #4774

feat: provide inline_transaction model for IO optimizing #4774

Uh oh!

majin1102 commented Sep 19, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Sep 19, 2025 •

edited

Loading

Uh oh!

jackye1995 Sep 19, 2025

Uh oh!

wjones127 Sep 19, 2025

Uh oh!

wjones127 left a comment

Uh oh!

wjones127 Sep 19, 2025

Uh oh!

wjones127 Sep 19, 2025

Uh oh!

wjones127 Sep 19, 2025

Uh oh!

majin1102 Sep 21, 2025

Uh oh!

wjones127 Sep 21, 2025

Uh oh!

jackye1995 Sep 21, 2025 •

edited

Loading

Uh oh!

majin1102 Sep 22, 2025

Uh oh!

wjones127 Sep 22, 2025

Uh oh!

jackye1995 Sep 22, 2025

Uh oh!

This comment was marked as outdated.

majin1102 commented Sep 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

	// If indices were also the last block, we can take the opportunity to
	// decode them now and cache them.
	if let Some(index_offset) = manifest.index_section {
	if manifest_size - index_offset <= last_block.len() {
	let offset_in_block = last_block.len() - (manifest_size - index_offset);
	let message_len =
	LittleEndian::read_u32(&last_block[offset_in_block..offset_in_block + 4])
	as usize;
	let message_data =
	&last_block[offset_in_block + 4..offset_in_block + 4 + message_len];
	let section = lance_table::format::pb::IndexSection::decode(message_data)?;
	let indices: Vec<Index> = section
	.indices
	.into_iter()
	.map(Index::try_from)
	.collect::<Result<Vec<_>>>()?;

	let ds_index_cache = session.index_cache.for_dataset(uri);
	let metadata_key = crate::session::index_caches::IndexMetadataKey {
	version: manifest_location.version,
	};
	ds_index_cache
	.insert_with_key(&metadata_key, Arc::new(indices))
	.await;
	}
	}

feat: provide inline_transaction model for IO optimizing #4774

Are you sure you want to change the base?

feat: provide inline_transaction model for IO optimizing #4774

Uh oh!

Conversation

majin1102 commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 Sep 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

majin1102 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

majin1102 commented Sep 19, 2025 •

edited

Loading

codecov-commenter commented Sep 19, 2025 •

edited

Loading

jackye1995 Sep 21, 2025 •

edited

Loading

majin1102 commented Sep 30, 2025 •

edited

Loading