The number of threads is retrieved from the thread pool after it's already been determined to be non-null. However, there's no check that getMaxThreads() returns a non-zero value before the check on line 1238-1240. If the pool was initialized with zero threads, this check would catch it, but the initialization logic should prevent this. Consider adding an assertion in the constructor where parse_pool_ptr is created to ensure parse_pool_size_ is non-zero when creating the pool.

The error message "zero number of threads" is unclear. It should specify which threads and where the configuration should be fixed. Consider changing to "Invalid configuration: async_insert_parse_threads must be greater than zero when parallel parsing is enabled".

I actually agree with Copilot on this one. The current message can be improved.

This is an impossible situation:
if the pool is created

 if (parse_pool_size_)
        parse_pool_ptr = std::make_shared<ThreadPool>(
            CurrentMetrics::AsynchronousInsertThreads,
            CurrentMetrics::AsynchronousInsertThreadsActive,
            CurrentMetrics::AsynchronousInsertThreadsScheduled,
            parse_pool_size_);

getMaxThreads cannot return zero.
May be assert is better than throwing LOGICAL_ERROR, but it is not a configuration error.

I don't think a LOGICAL_ERROR is a bad choice, but the message can be polished

The comment "it seems that we cannot share format between threads" is uncertain and doesn't explain why. Either confirm this limitation with a definitive statement or explain the underlying reason (e.g., thread-safety issues, mutable state in format objects).

The chunks.resize(num_threads) on line 1377 is redundant because chunks is already initialized with size num_threads on line 1111. This resize could be removed to avoid confusion.

Agree wth Copilot on this one

This has big impact.
Before there was one chunk, now there are many chunks.
Before it has been inserted as one part in the landing table,
now it could be as many parts as chunks count here.

That's a very good concern, thanks.
I'll recheck if it is true, and if it is, rollback to a previous version where I joined chunks explicitly.

Well,
this is true,
except SquashingTransform, which is controlled by Setting::min_insert_block_size_rows & Co.
(see InterpreterInsertQuery::buildInsertPipeline).

Plan to forcibly set these parameters to maximum.

Seems slightly better than doing basically that same in AsynchronousInsertQueue::processData .

WDYT?

It is not good to pin any settings in the code.
I understand that those chunks could be squashed together later. But this is not guaranteed. This is my concern. It was only one chunk from async insert before. Now it could be several. And most likely there are no tests in CI for the cases with several chunks.

What is wrong with AsynchronousInsertQueue::processData?

What is wrong with AsynchronousInsertQueue::processData?

Do you suggest joining parts in AsynchronousInsertQueue::processData ?
Could you provide more reasons/hints?

Honestly I don't see why it is a good idea to duplicate this code. We use INSERT pipeline and SquashingTransform is a feature of it.

most likely there are no tests in CI for the cases with several chunks.

I am adding validation of number of parts to the test.

Parsing data during async insert part is rather tricky, because we cannot afford creating format object every time, we have to maintain a pool.

I don't have a clear opinion towards this approach. It is natural and tempting.
Actually it is what I started from, but then I switched to what we currently have in the branch because it solves the problem (reduces latency) and just works.

we cannot afford creating format object every time

Do not understand this.

I am used to thinking that constructors for some format objects are expensive, that's why creating format for every async insert is not an option.

Does it make sense?

I do not understand, why is it expensive?

I do not understand, why is it expensive?

It is expensive for AvroConfluentRowInputFormat because it requires network interactions, it is expensive for protobuf because proto file must be read (I think).

logging, timeit and floor are unused, can be removed

thread_num is never used

Why a lambda here instead of defining a new function (even if it's within _generate_values)

This code is borrowed from another test, specifically test_async_insert_adaptive_busy_timeout

I can improve it though if you think that it makes sense.

Please add 2 blank spaces before every function definition

Why not using an f-string instead of format?

Why not using an f-string instead of format?

Yes, switching to f-string.

I am still struggling with joining parts, will push all changes together.

What's the end purpose of initializing this here of it's assigned afterwards?

This is effectively the same as to assign to chunks[0].
To me explicit reference 'chunk' adds some verbosity, though I can get rid of it if you think that it is worth doing.

I don't think a LOGICAL_ERROR is a bad choice, but the message can be polished

Why does this need to be a shared_ptr?

Is it ok to assume here chunks[0] exists? Is it because this is past the num_rows == 0 check?

Agree wth Copilot on this one