Data compression between the CH servers and the drivers #523

ilejn · 2025-08-27T16:48:20Z

Addresses #514

ilejn · 2025-08-29T22:49:42Z

I wonder if ODBC driver should send enable_http_compression=1 URL parameter. It is required to get compressed response, Content-Encoding it is not enough.

Probably yes, assuming compression is a useful feature to be enabled by default. It is cumbersome to explicitly set enable_http_compression as a session parameter or in users.xml .
While what is the actual purpose of enable_http_compression setting?

slabko · 2025-09-09T10:47:02Z

Hi @ilejn,

There’s an ongoing discussion about enable_http_compression in #71591, which explains why this setting exists. It also suggests that a better compression method is likely needed, since the setting was specifically introduced to avoid using gzip and deflate.

Regarding enabling it by default, I’d hold off for now, since this is going to be a new feature and needs to be tested better first.

ilejn · 2025-09-09T11:55:47Z

Hi @slabko ,
thank you for the clarification.

gzip (and deflate) support is straightforward — using Poco’s InflatingInputStream as a wrapper is sufficient.

I will investigate LZ4 as well.

If successful, I plan to enable the Compression configuration flag by default. While enabled, the driver will send an Accept-Encoding header with the value "lz4, gzip, deflate" and set the enable_http_compression=1 parameter.
If LZ4 support proves too complex (which is unlikely) and we remain limited to gzip, the Compression flag will be disabled by default. In that case, the driver will send Accept-Encoding: "gzip, deflate" and set enable_http_compression=0.

So I plan to send the Accept-Encoding header in all cases.

Does this approach make sense?

slabko · 2025-09-09T18:44:31Z

@ilejn

Let's not enable it by default yet, I would keep it as a parameter for now, disabled by default. We can enable it by default later, after people have had a chance to try it. I understand your logic, but I don’t fully trust some parts of the implementation, especially Poco. I’ve already encountered a number of problems with it. Am I missing any reason we’d want it on by default from the get-go?

Everything else, however, makes complete sense. We need to enable enable_http_compression=1. In fact, we could even enable it always, if we want, and then pass Accept-Encoding based on the configuration parameter.

One more thing that might help: we are planning to update Poco to a more recent version. So if the current version doesn’t support LZ4, you can check it the newer version, since we’re going to use the new version soon.

ilejn · 2025-09-09T20:25:35Z

Ok, lets keep compression disabled regardless of LZ4. No special reasons to have it enabled by default.

To me passing Accept-Encoding unconditionally looks slightly better (than passing enable_http_compression) e.g. because it allows to forcibly enable compression using server side settings and because enable_http_compression without Accept-Encoding seems a bit peculiar.

Unfortunately there is no signs of LZ4 in recent Poco.
I am borrowing implementation from DB::Lz4InflatingReadBuffer

windsurf-bot

Other comments (14)

driver/format/ODBCDriver2.h (64-64) The parameter type for `raw_stream` was changed from a reference to a pointer. Make sure there's appropriate null checking in the implementation to prevent potential null pointer dereferences.

driver/lz4_inflating_input_stream.cpp (44-44) Missing closing parenthesis in the throw statement.

                    throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk)"));

driver/result_set.h (141-141) The constructor parameter has been changed from a reference (`std::istream &`) to a pointer (`std::istream *`). This introduces the possibility of null pointer dereferences. Make sure the implementation includes null checks before using this pointer.
driver/result_set.cpp (347-347) Using STREAM_GZIP for both 'gzip' and 'deflate' compression types may not be correct. Poco::InflatingStreamBuf provides different stream types (STREAM_GZIP, STREAM_ZLIB, STREAM_DEFLATE). Consider using the appropriate type for each compression format.
driver/lz4_inflating_input_stream.h (19-19) The `reset()` method is empty but appears in the public interface. If this is intended to reset the decompression state, it should properly reset internal state variables like `bytes_left`, `need_more_input`, and potentially the LZ4 context. Otherwise, it might be misleading to users of this class.
driver/connection.cpp (453-453) This commented out LOG statement appears to be debugging code that was left in the PR. Consider removing it before merging.
packaging/odbc.ini.sample (45-45) There's a potential confusion in the comment for the Compress option. When Compress is set to 'off' (the default), would it still pass enable_http_compression=1 to the server? The comment should clarify whether 'off' means no compression or if it means the parameter isn't passed at all.
driver/format/RowBinaryWithNamesAndTypes.cpp (317-324) There's significant code duplication between the two constructors. Consider refactoring to have one constructor call the other after setting up the base class, or extract the common initialization logic into a private helper method.
driver/result_set.cpp (343-343) Using raw pointers for stream management could lead to potential issues. Consider using a reference or smart pointer to make ownership clearer. If raw_stream's lifetime is guaranteed to exceed the ResultReader's lifetime, document this assumption.
driver/lz4_inflating_input_stream.h (29-29) The `src_buffer` is declared as a raw pointer without clear ownership semantics. Consider using a smart pointer or including it in the vector to avoid potential memory leaks or undefined behavior.

driver/lz4_inflating_input_stream.cpp (69-69) Missing space between version and error message.

                std::string("LZ4 decompression failed. LZ4F version: ") + std::to_string(LZ4F_VERSION) + " Error: " + LZ4F_getErrorName(ret));

driver/lz4_inflating_input_stream.h (43-43) There's a typo in the comment: "Exacltly" should be "Exactly".
```
/// Exactly the same as Poco::InflatingIOS and Poco:InflatingInputStream
```

driver/result_set.h (141-142) There's an indentation inconsistency in the second constructor declaration. It should be aligned with the first constructor for better readability.

    explicit ResultReader(const std::string & timezone_, std::istream * stream, std::unique_ptr<ResultMutator> && mutator);
    explicit ResultReader(const std::string & timezone_, std::istream * stream, std::unique_ptr<ResultMutator> && mutator, std::unique_ptr<std::istream> && inflating_input_stream);

driver/result_set.cpp (315-315) The parameter comment suggests a specific type (Poco::InflatingInputStream) but the actual type is more generic (std::unique_ptr). Consider either removing the comment or making the type more specific if that's the intended use.

💡 To request another review, post a new comment with "/windsurf-review".

windsurf-bot · 2025-09-19T09:58:00Z

driver/utils/amortized_istream_reader.h

+            raw_stream_->read(&buffer_[offset_ + avail], to_read);

-            if (raw_stream_.gcount() < to_read)
-                buffer_.resize(buffer_.size() - (to_read - raw_stream_.gcount()));
+            if (raw_stream_->gcount() < to_read)
+                buffer_.resize(buffer_.size() - (to_read - raw_stream_->gcount()));


The tryPrepare method should check if raw_stream_ is null before attempting to read from it.

Suggested change

raw_stream_->read(&buffer_[offset_ + avail], to_read);

if (raw_stream_.gcount() < to_read)

buffer_.resize(buffer_.size() - (to_read - raw_stream_.gcount()));

if (raw_stream_->gcount() < to_read)

buffer_.resize(buffer_.size() - (to_read - raw_stream_->gcount()));

raw_stream_->read(&buffer_[offset_ + avail], to_read);

if (!raw_stream_ || raw_stream_->gcount() < to_read)

buffer_.resize(buffer_.size() - (to_read - raw_stream_->gcount()));

~~Not sure why this code is relevant~~ Addressed.

windsurf-bot · 2025-09-19T09:58:00Z

driver/connection.cpp

+    if (enable_http_compression && !enable_http_compression_set) {
+        uri.addQueryParameter("enable_http_compression", enable_http_compression ? "1" : "0");
+    }


The current implementation only adds the enable_http_compression parameter to the URI when it's set to true. This means if compression is enabled by default on the server side and you want to explicitly disable it, setting enable_http_compression=0 in the connection string won't have any effect. Consider always adding the parameter when it's explicitly set (regardless of value):

Suggested change

if (enable_http_compression && !enable_http_compression_set) {

uri.addQueryParameter("enable_http_compression", enable_http_compression ? "1" : "0");

}

if (!enable_http_compression_set) {

uri.addQueryParameter("enable_http_compression", enable_http_compression ? "1" : "0");

}

Yes, 'if compression is enabled by default on the server side' it is fine, data will be compressed.

windsurf-bot · 2025-09-19T09:58:00Z

driver/format/RowBinaryWithNamesAndTypes.h

 {
 public:
-    explicit RowBinaryWithNamesAndTypesResultReader(const std::string & timezone, std::istream & raw_stream, std::unique_ptr<ResultMutator> && mutator);
+    explicit RowBinaryWithNamesAndTypesResultReader(const std::string & timezone, std::istream * raw_stream, std::unique_ptr<ResultMutator> && mutator);


The parameter type has been changed from std::istream & to std::istream *. Make sure to add null pointer checks in the implementation to prevent potential crashes.

ilejn · 2025-09-25T07:00:03Z

/windsurf-review

windsurf-bot

Other comments (12)

driver/format/RowBinaryWithNamesAndTypes.cpp (308-308) The constructor now takes a pointer to `std::istream` instead of a reference, but there's no null check before using this pointer. Consider adding a null check to prevent potential crashes if a null pointer is passed.
driver/utils/amortized_istream_reader.h (43-43) The eof() method doesn't check if raw_stream_ is null before dereferencing it. This could lead to a segmentation fault.
```
if (!raw_stream_ || raw_stream_->eof() || raw_stream_->fail())
    return true;
```
driver/utils/amortized_istream_reader.h (51-51) The second check in the eof() method also needs to verify if raw_stream_ is null before dereferencing it.
```
return (!raw_stream_ || raw_stream_->eof() || raw_stream_->fail());
```
driver/format/RowBinaryWithNamesAndTypes.h (78-78) The parameter type has been changed from `std::istream &` to `std::istream *`. Make sure to add null checks in the implementation to prevent potential null pointer dereferences.
driver/lz4_inflating_input_stream.h (19-19) The `reset()` method is empty but doesn't reset the internal state variables like `bytes_left`, `need_more_input`, or the LZ4 decompression context. If this method is intended to reset the stream buffer to its initial state, consider properly resetting these variables.

driver/result_set.cpp (346-354) The code doesn't handle the case where an unsupported compression format is specified. Consider adding validation or at least a warning when an unrecognized compression format is provided.

    if (compression == "gzip" || compression == "deflate") {
        inflating_input_stream = make_unique<Poco::InflatingInputStream>(raw_stream, Poco::InflatingStreamBuf::STREAM_GZIP);
        stream_ptr = inflating_input_stream.get();
    } else if (compression == "lz4") {
        inflating_input_stream = make_unique<LZ4InflatingInputStream>(raw_stream);
        stream_ptr = inflating_input_stream.get();
    } else if (!compression.empty() && compression != "none") {
        throw std::runtime_error("'" + compression + "' compression format is not supported");
    } else {
        stream_ptr = &raw_stream;
    }

driver/result_set.h (141-142) The ResultReader constructor has been changed to take a pointer to std::istream instead of a reference. This introduces the possibility of null pointer dereference. Consider adding null checks in the constructor implementation or using std::reference_wrapper if a nullable reference is needed.
driver/lz4_inflating_input_stream.h (29-29) The `src_buffer` is declared as a raw pointer but there's no visible allocation/deallocation strategy in this header. Ensure this pointer is properly managed in the implementation to prevent memory leaks.

driver/lz4_inflating_input_stream.cpp (69-69) Missing space in error message between version and 'Error:'.

                std::string("LZ4 decompression failed. LZ4F version: ") + std::to_string(LZ4F_VERSION) + " Error: " + LZ4F_getErrorName(ret));

driver/connection.cpp (453-453) This debug log statement should be removed before merging.
driver/lz4_inflating_input_stream.h (43-43) Typo in comment: "Exacltly" should be "Exactly".
driver/result_set.h (162-165) Consider using a more consistent formatting style for the make_result_reader function declaration. Either keep it on a single line or align parameters more consistently for better readability.

💡 To request another review, post a new comment with "/windsurf-review".

windsurf-bot · 2025-09-25T07:04:02Z

driver/lz4_inflating_input_stream.cpp

+            if (need_more_input) {
+                assert(bytes_left);
+                if (compressed_buffer.size() <= bytes_left) {
+                    throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk"));


Missing closing parenthesis in the throw statement.

Suggested change

throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk"));

throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk)"));

windsurf-bot · 2025-09-25T07:04:02Z

driver/connection.cpp

            valid_value = (value.empty() || isYesOrNo(value));
            if (valid_value) {
-                auto_session_id = isYes(value);
+                auto_session_id = (value.empty() || isYes(value));


This changes the behavior of auto_session_id to treat empty values as 'yes'. This is a behavior change that should be documented in the PR description or release notes.

windsurf-bot · 2025-09-25T07:04:02Z

driver/format/ODBCDriver2.cpp

 }

-ODBCDriver2ResultReader::ODBCDriver2ResultReader(const std::string & timezone_, std::istream & raw_stream, std::unique_ptr<ResultMutator> && mutator)
+ODBCDriver2ResultReader::ODBCDriver2ResultReader(const std::string & timezone_, std::istream * raw_stream, std::unique_ptr<ResultMutator> && mutator)


The constructor now accepts a pointer std::istream * instead of a reference, but there's no null check before using it. This could lead to null pointer dereference if a null pointer is passed.

It is checked in ResultReader, should be enough.

ilejn · 2025-09-25T07:32:31Z

/windsurf-review

windsurf-bot

Other comments (15)

driver/format/RowBinaryWithNamesAndTypes.cpp (308-309) The constructor now takes a pointer instead of a reference, but there's no null check before using `raw_stream` in the constructor body. This could lead to a null pointer dereference if null is passed.
driver/format/RowBinaryWithNamesAndTypes.cpp (317-318) Similar to the first constructor, there's no null check for the `raw_stream` pointer before it's used in this new constructor.
driver/lz4_inflating_input_stream.cpp (9-18) The class member variables 'bytes_left', 'src_buffer', and 'need_more_input' are used but not initialized in the constructor. This could lead to undefined behavior when first accessing these variables.
driver/lz4_inflating_input_stream.h (19-19) The `reset()` method is empty but doesn't reset the internal state variables like `bytes_left`, `need_more_input`, or the LZ4 decompression context. If this method is intended to reset the stream buffer for reuse, it should properly reset all internal state.
driver/lz4_inflating_input_stream.h (35-35) The `LZ4F_dctx` pointer needs proper cleanup in the destructor using `LZ4F_freeDecompressionContext()`. Ensure this is handled in the implementation file to prevent memory leaks.
driver/result_set.h (141-142) The ResultReader constructor was changed to take a pointer to std::istream instead of a reference. This introduces the possibility of null pointer dereference. Consider adding null checks or using a reference if the stream is guaranteed to exist.
driver/lz4_inflating_input_stream.cpp (68-69) Consider using a string stream for constructing the error message instead of multiple string concatenations, which can be inefficient and harder to maintain.

driver/result_set.cpp (346-354) Consider explicitly handling the case when compression is empty to make the code more readable:

    if (compression == "gzip" || compression == "deflate") {
        inflating_input_stream = make_unique<Poco::InflatingInputStream>(raw_stream, Poco::InflatingStreamBuf::STREAM_GZIP);
        stream_ptr = inflating_input_stream.get();
    } else if (compression == "lz4") {
        inflating_input_stream = make_unique<LZ4InflatingInputStream>(raw_stream);
        stream_ptr = inflating_input_stream.get();
    } else if (compression.empty()) {
        stream_ptr = &raw_stream;
    } else {
        stream_ptr = &raw_stream;
    }

driver/lz4_inflating_input_stream.h (29-29) The `src_buffer` is declared as a raw pointer without clear ownership semantics. Consider using a smart pointer or ensuring proper allocation/deallocation in the constructor/destructor to prevent memory leaks.
driver/connection.cpp (433-433) The condition for setting `enable_http_compression` only checks for positive cases (`typed_value == 1 || isYes(value)`), but doesn't explicitly handle the case when `typed_value == 0 && isNo(value)`. For consistency with other similar code in this file, consider updating to:
```
                enable_http_compression = (typed_value == 1 || isYes(value)) && !(typed_value == 0 || isNo(value));
```
driver/format/RowBinaryWithNamesAndTypes.h (78-78) The parameter type has been changed from `std::istream &` to `std::istream *`. If null values are allowed, this should be documented with a comment explaining how null values are handled.
driver/result_set.h (162-165) The make_result_reader function now includes a compression parameter, but there's no documentation about what compression formats are supported. Consider adding comments or documentation about the supported compression formats and their expected values.

driver/lz4_inflating_input_stream.cpp (69-69) There's a missing space between the version number and 'Error:' text in the error message.

                std::string("LZ4 decompression failed. LZ4F version:") + std::to_string(LZ4F_VERSION) + " Error: " + LZ4F_getErrorName(ret));

driver/lz4_inflating_input_stream.h (43-43) Typo in comment: "Exacltly" should be "Exactly".
driver/connection.cpp (453-453) This commented out LOG statement appears to be debugging code that was left in. Consider removing it before merging.

💡 To request another review, post a new comment with "/windsurf-review".

windsurf-bot · 2025-09-25T07:36:32Z

driver/utils/amortized_istream_reader.h

+    explicit AmortizedIStreamReader(std::istream * raw_stream)
        : raw_stream_(raw_stream)
    {


The constructor now accepts a null pointer, but there's no validation in the constructor itself. Consider adding a null check in the constructor or documenting that null is a valid value.

windsurf-bot · 2025-09-25T07:36:32Z

driver/utils/amortized_istream_reader.h

        // (it should be done in reverse order)
        if (available() > 0) {
            for (auto it = buffer_.rbegin(); it < buffer_.rend() - offset_; ++it) {
-                raw_stream_.putback(*it);
+                raw_stream_->putback(*it);
            }
        }


The destructor attempts to putback characters to the stream without checking if raw_stream_ is null, which could lead to a null pointer dereference.

windsurf-bot · 2025-09-25T07:36:32Z

driver/format/ODBCDriver2.h

 {
 public:
-    explicit ODBCDriver2ResultReader(const std::string & timezone_, std::istream & raw_stream, std::unique_ptr<ResultMutator> && mutator);
+    explicit ODBCDriver2ResultReader(const std::string & timezone_, std::istream * raw_stream, std::unique_ptr<ResultMutator> && mutator);


Changing the raw_stream parameter from a reference to a pointer introduces the possibility of null pointer dereferences. Make sure the implementation includes null checks before using this pointer.

ilejn · 2025-09-25T08:10:01Z

Hello Andrew @slabko ,
could you have a look, please.

slabko · 2025-09-29T13:27:28Z

@ilejn

There seems to be some memory corruption. Could you please check what is going on? I suggest checking the output of the sanitizer actions.

ilejn · 2025-09-29T21:30:22Z

I believe that the issue should be fixed by

--- a/driver/utils/amortized_istream_reader.h
+++ b/driver/utils/amortized_istream_reader.h
@@ -128,7 +128,7 @@ private:
     }
 
 private:
-    std::istream * raw_stream_;
+    std::istream * raw_stream_ = 0;

I'll go throw the code again and may be switch to unique_ptr.
Initially it seemed redundant/pointless to me, but now I am not that sure.

ilejn · 2025-09-30T10:53:43Z

I have issues with sanitizer builds (because of some reasons isql does not work if the driver built with sanitizer),
but the issue should be fixed.

slabko · 2025-09-30T15:14:39Z

I have issues with sanitizer builds (because of some reasons isql does not work if the driver built with sanitizer),
but the issue should be fixed.

This is a problem, yes. This is because everything should be build with the sanitizers enabled, including isql and unixODBC.
But you can still run the tests, they use unixODBC built together with the driver.

Can you try this configuration:
https://github.com/ClickHouse/clickhouse-odbc/blob/master/.github/workflows/Sanitizer.yml#L66

ilejn · 2025-10-01T14:13:45Z

Memory corruption is gone.

I presume that Windows and macOS build errors are not related

  Generating code
  Previous IPDB not found, fall back to full compilation.
  All 174 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
  Finished generating code
  clickhouse-odbcw-load-ut.vcxproj -> D:\a\clickhouse-odbc\clickhouse-odbc\build\driver\test\Release\clickhouse-odbcw-load-ut.exe
Error: Process completed with exit code 1.

are they ?

traceon · 2025-10-01T17:03:54Z

driver/CMakeLists.txt

    object.h
    result_set.h
    statement.h
+    lz4_inflating_input_stream.h


This brings lz4 lib dependency to the interface... You need to explicitly depend on it now. (Ideally, you could hide it inside .cpp.)

Hello @traceon,
this is doable

--- a/driver/CMakeLists.txt +++ b/driver/CMakeLists.txt @@ -120,7 +120,7 @@ add_library (${libname}-impl STATIC object.h result_set.h statement.h - lz4_inflating_input_stream.h + # lz4_inflating_input_stream.h ) if (CH_ODBC_ENABLE_CODE_COVERAGE) diff --git a/driver/lz4_inflating_input_stream.cpp b/driver/lz4_inflating_input_stream.cpp index 6918c1a..abbd662 100644 --- a/driver/lz4_inflating_input_stream.cpp +++ b/driver/lz4_inflating_input_stream.cpp @@ -1,4 +1,6 @@ #include "lz4_inflating_input_stream.h" +#include <lz4.h> +#include <lz4frame.h> #include <assert.h> #include "Poco/Exception.h" diff --git a/driver/lz4_inflating_input_stream.h b/driver/lz4_inflating_input_stream.h index 4515217..f168002 100644 --- a/driver/lz4_inflating_input_stream.h +++ b/driver/lz4_inflating_input_stream.h @@ -3,8 +3,15 @@ #include <istream> #include "Poco/BufferedStreamBuf.h" -#include <lz4.h> -#include <lz4frame.h> +// #include <lz4.h> +// #include <lz4frame.h> + +extern "C" +{ + typedef struct LZ4F_dctx_s LZ4F_dctx; +} +

But why? Do we have particular concerns about the dependency on lz4 (besides general intention to cut unnecessary dependencies)?

Honestly I am not sure that it would be an improvement.

And (correct me if I am wrong) listing a header in add_library is meaningless until the library is installable.

Saying it is public is probably overstatement here, but the fact that win builds are failing not founding lz4.h means some missed listed deps. Unix build pass probably because the system lz4.h is picked up, which is undesirable. So the proper solution would be making sure the variant from contrib is picked.

Thank you for pointing this out

D:\a\clickhouse-odbc\clickhouse-odbc\source\driver\lz4_inflating_input_stream.h(6,10): error C1083: Cannot open include file: 'lz4.h': No such file or directory [D:\a\clickhouse-odbc\clickhouse-odbc\build\driver\clickhouse-odbcw-impl.vcxproj]

I somehow missed it.

So saying 'presume that Windows and macOS build errors are not related' I was wrong.

I think that the real problem is lz4 is not properly imported. Will fix it.

ilejn · 2025-10-02T12:54:29Z

Hello @slabko ,
I've included header missed on Mac and Windows.
If it is not the latest problem will create local Windows environment and play with it.

ilejn · 2025-10-02T14:29:58Z

Checked with VS 2022.
Built OK.

ilejn · 2025-10-08T13:00:10Z

Hello @slabko ,
further suggestions or concerns to address?

slabko · 2025-10-08T13:59:18Z

Hi @ilejn

I can see that it builds and passes the tests. I’m also working on some network-related issues, such as updating Poco. As soon as I have a bit more time, I’ll take a look at your PR.

Thank you very much for the contribution!

Enmk · 2025-10-08T16:44:09Z

driver/connection.cpp

                client_name = value;
            }
        }
+        else if (Poco::UTF8::icompare(key, INI_COMPRESS) == 0 || Poco::UTF8::icompare(key, INI_USE_COMPRESSION) == 0) {


Do I get it right that:

"Compress" and "UseCompression" are synonymous? (
If yes, then please add that to the docs.

having any of those keys in the config without actual value equals to "NO" ?
If yes, then that is somewhat frustrating, and has to be either fixed or clearly specified in docs.

Enmk · 2025-10-08T17:03:26Z

driver/result_set.cpp

+        inflating_input_stream = make_unique<LZ4InflatingInputStream>(raw_stream);
+        stream_ptr = inflating_input_stream.get();
+    } else {
+        stream_ptr = &raw_stream;


just in case, if remote party sends compression="foobar", then it is silently assumed to be "no compression"... I suggest either throwing an error or at least writing a warning to a log.

Enmk · 2025-10-08T17:14:39Z

driver/result_set.cpp

+        std::istream & raw_stream,
+        std::unique_ptr<ResultMutator> && mutator) {
+    std::istream * stream_ptr = nullptr;
+    std::unique_ptr<std::istream> inflating_input_stream;


Also, this ownership model seems a bit clumsy, maybe it would be worth moving ownership of the wrapping stream to the AmortizedIStreamReader and pass it to the ResultReader ? On the downside that would make AmortizedIStreamReader a tiny bit more complex.

Well spotted, ownership model is proved to be error prone.
It is implemented in this way because ResultReader is a kind of wrapper around AmortizedIStreamReader and acts as a resource holder and because I tried to keep AmortizedIStreamReader intact.

I moved holder from ResultReader to AmortizedIStreamReader.
Besides this I switched back to reference for raw_stream_ member of AmortizedIStreamReader. It does not change anything, just reduces number of modification.

Enmk

Mostly Ok, please fix minor issues mentioned in comments

ilejn added 2 commits August 27, 2025 19:43

support_compression: initial

24b735f

support_compression: slightly works

6a8badc

ilejn added 5 commits September 17, 2025 12:13

support_compression: LZ4 seems to be working

7b92ea0

support_compression: code cleanup

da17f18

support_compression: RowBinaryWithNamesAndTypes, tiny cleanup

a089e5d

support_compression: some doc, test

64eab4d

support_compression: need_more_input + minor improvements

347db71

ilejn marked this pull request as ready for review September 19, 2025 09:53

windsurf-bot bot reviewed Sep 19, 2025

View reviewed changes

support_compression: minor things per windsurf-bot' review

937b78f

windsurf-bot bot reviewed Sep 25, 2025

View reviewed changes

support_compression: irrelevant session_id change reverted, minor

db61e10

windsurf-bot bot reviewed Sep 25, 2025

View reviewed changes

support_compression: one more null pointer check

b31874f

support_compression: destruction order fix (sanitizer complain)

b15b466

traceon reviewed Oct 1, 2025

View reviewed changes

support_compression: add lz4 library

19a0829

support_compression: include vector (for Win and mac)

9e98da6

support_compression: tiny cleanup

638008d

Enmk reviewed Oct 8, 2025

View reviewed changes

support_compression: stream owning model modified

8c82dc3

	throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk"));
	throw Poco::IOException(std::string("LZ4 decompression failed, internal error (not enough room to create data chunk)"));

Data compression between the CH servers and the drivers #523

Are you sure you want to change the base?

Data compression between the CH servers and the drivers #523

Uh oh!

Conversation

ilejn commented Aug 27, 2025

Uh oh!

ilejn commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slabko commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn commented Sep 9, 2025

Uh oh!

slabko commented Sep 9, 2025

Uh oh!

ilejn commented Sep 9, 2025

Uh oh!

windsurf-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn commented Sep 25, 2025

Uh oh!

windsurf-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilejn commented Sep 25, 2025

Uh oh!

windsurf-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

windsurf-bot bot Sep 25, 2025

Choose a reason for hiding this comment

Uh oh!

ilejn commented Sep 25, 2025

Uh oh!

slabko commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn commented Sep 30, 2025

Uh oh!

slabko commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilejn commented Aug 29, 2025 •

edited

Loading

slabko commented Sep 9, 2025 •

edited

Loading

ilejn Sep 19, 2025 •

edited

Loading

slabko commented Sep 29, 2025 •

edited

Loading

ilejn commented Sep 29, 2025 •

edited

Loading

slabko commented Sep 30, 2025 •

edited

Loading

ilejn commented Oct 1, 2025 •

edited

Loading

ilejn Oct 1, 2025 •

edited

Loading