Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work) #33957

azat · 2022-01-24T16:06:35Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fix getauxval() in glibc-compatibility, this should fix vsyscalls after setenv (i.e. timezone is set in config), and LSan (and also fix some leaks that had been found by LSan)

getauxval() from glibc-compatibility did not work always correctly:

it does not work after setenv(), and this breaks vsyscalls,
like sched_getcpu() 1 (and BaseDaemon.cpp always set TZ if timezone
is defined, which is true for CI 2).
another think that is definitely broken is LSan (Leak Sanitizer), it
relies on worked getauxval() but it does not work if __environ is not
initialized yet (there is even a commit about this).

And because of at least one issue hadn't been catched before keeper: fix memory leak in case of compression is used (default) #33840

Fix this by using /proc/self/auxv.

And let's see how many issues will LSan find...

I've verified this patch manually by printing AT_BASE and compared it
with output of LD_SHOW_AUXV.

Cc: @alesapin
Cc: @filimonov (#27492)
Cc: @alexey-milovidov (#28132)
Cc: @vitlibar (#15111)

base/glibc-compatibility/musl/getauxval.c

alexey-milovidov · 2022-01-24T18:06:20Z

What are the exact cases when it does not work?
Simple test
sudo strace -f -e trace=clock_gettime,gettimeofday -p $(pidof clickhouse-server)
does not show these syscalls.

alexey-milovidov · 2022-01-24T18:09:25Z

Fix bug which lead to memory leak ClickHouse/NuRaft#36

This is actually a logical leak like unbounded container growth.

azat · 2022-01-24T19:02:41Z

This is actually a logical leak like unbounded container growth.

Right.

What are the exact cases when it does not work?

This does not means that those syscalls will not work, but instead this means that invalid memory access will be if vsyscall was not initialized.

The problem is that after setenv you cannot rely on that auxiliary vector lies after __environ because setenv will do realloc.

I've tried to reproduce the problem with clock_gettime, but failed. I guess the reason is that it is called before first setenv, and so pointer already changed (and so it does not requires access to AT_SYSINFO_EHDR)

I saw this issue in #32928 where rocksdb as a cache layer for MergeTree metadata had been introduced, and rocksdb requires sched_getcpu() which fails - #32928 (comment)

alexey-milovidov · 2022-01-24T19:22:23Z

Yes, last time I checked, ClickHouse did not work without /proc on Linux.

azat · 2022-01-25T19:53:05Z

Now LSan looks happy

base/glibc-compatibility/musl/getauxval.c

alexey-milovidov · 2022-01-26T07:22:53Z

I also appreciate any help with #31833

We will be able to build clickhouse binary with Musl and clickhouse-odbc-bridge, clickhouse-library-bridge separately with glibc and with mininum dependencies.

getauxval() from glibc-compatibility did not work always correctly: - it does not work after setenv(), and this breaks vsyscalls, like sched_getcpu() [1] (and BaseDaemon.cpp always set TZ if timezone is defined, which is true for CI [2]). [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1163404 [2]: ClickHouse#32928 (comment) - another think that is definitely broken is LSan (Leak Sanitizer), it relies on worked getauxval() but it does not work if __environ is not initialized yet (there is even a commit about this). And because of, at least, one leak had been introduced [3]: [3]: ClickHouse#33840 Fix this by using /proc/self/auxv. And let's see how many issues will LSan find... I've verified this patch manually by printing AT_BASE and compared it with output of LD_SHOW_AUXV. Signed-off-by: Azat Khuzhin <[email protected]>

CI founds after LSan had been fixed [1]: 01889_sqlite_read_write: [ FAIL ] 8.32 sec. - return code: 1 ================================================================= ==20649==ERROR: LeakSanitizer: detected memory leaks Indirect leak of 1968 byte(s) in 1 object(s) allocated from: 0 0xc5c1ffd in operator new(unsigned long) (/usr/bin/clickhouse+0xc5c1ffd) 1 0x25e32d0d in std::__1::__unique_if<DB::StorageInMemoryMetadata>::__unique_single std::__1::make_unique<DB::StorageInMemoryMetadata, DB::StorageInMemoryMetadata const&>(DB::StorageInMemoryMetadata c> 2 0x25e32d0d in DB::IStorage::setInMemoryMetadata(DB::StorageInMemoryMetadata const&) obj-x86_64-linux-gnu/../src/Storages/IStorage.h:194:22 3 0x29bdee98 in DB::StorageSQLite::StorageSQLite(DB::StorageID const&, std::__1::shared_ptr<sqlite3>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std:> 4 0x25ee61d6 in std::__1::shared_ptr<DB::StorageSQLite> shared_ptr_helper<DB::StorageSQLite>::create<DB::StorageID, std::__1::shared_ptr<sqlite3> const&, std::__1::basic_string<char, std::__1::char_tr> 5 0x25ee61d6 in DB::TableFunctionSQLite::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1:> SUMMARY: AddressSanitizer: 171256 byte(s) leaked in 130 allocation(s). [1]: https://github.com/ClickHouse/ClickHouse/runs/4929706698?check_suite_focus=true Signed-off-by: Azat Khuzhin <[email protected]>

azat · 2022-01-27T08:07:37Z

Performance Comparison (actions) [1/4] — Failed to parse the report.

curl: (6) Could not resolve host: s3.amazonaws.com

Integration tests (release, actions) [2/2] — fail: 105, passed: 0, flaky: 0

ERROR: for zoo2  Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

E               ERROR: for node1  Get "https://registry-1.docker.io/v2/": context deadline exceeded
E               Get "https://registry-1.docker.io/v2/": context deadline exceeded

Stress test (address, actions) — Sanitizer assert (in stderr.log)

Interesting, more leaks had been found, but there were no leaks in the previous run, will take a look.

LSan found [1]: Direct leak of 5170176 byte(s) in 5049 object(s) allocated from: 0 0xc598edd in malloc (/usr/bin/clickhouse+0xc598edd) 1 0x39679739 in (anonymous namespace)::itanium_demangle::initializeOutputStream(char*, unsigned long*, (anonymous namespace)::itanium_demangle::OutputStream&, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxxabi/src/demangle/Utility.h:178:31 2 0x39679739 in __cxa_demangle obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_demangle.cpp:351:13 3 0x28f6f3ed in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:662:44 [1]: https://s3.amazonaws.com/clickhouse-test-reports/33957/08f4f45fd9da923ae3e3fdd8a527c297d35247eb/stress_test__address__actions_.html Signed-off-by: Azat Khuzhin <[email protected]>

CI reports [1]: Indirect leak of 648 byte(s) in 9 object(s) allocated from: ... 2 0x12b96503 in DB::AggregateFunctionSimpleState::getReturnType() const obj-x86_64-linux-gnu/../src/AggregateFunctions/AggregateFunctionSimpleState.h:47:15 ... [1]: https://s3.amazonaws.com/clickhouse-test-reports/33957/08f4f45fd9da923ae3e3fdd8a527c297d35247eb/stress_test__address__actions_.html After we can get this query by using query_log artifact: $ wget https://s3.amazonaws.com/clickhouse-test-reports/33957/08f4f45fd9da923ae3e3fdd8a527c297d35247eb/stress_test__address__actions_/query_log_dump.tar $ tar -xf query_log_dump.tar $ clickhouse-local --path var/lib/clickhouse/ SELECT query FROM system.query_log ARRAY JOIN used_aggregate_function_combinators AS func WHERE has(used_aggregate_functions, 'groupBitOr') AND has(used_aggregate_function_combinators, 'SimpleState') AND (type != 'QueryStart') Query id: 5b7722b3-f77e-4e7e-bd0b-586d6d32a899 ┌─query────────────────────────────────────────────────────────────────────────────┐ │ with groupBitOrSimpleState(number) as c select toTypeName(c), c from numbers(1); │ └──────────────────────────────────────────────────────────────────────────────────┘ Fixes: 01570_aggregator_combinator_simple_state.sql Fixes: ClickHouse#16853 Signed-off-by: Azat Khuzhin <[email protected]>

Signed-off-by: Azat Khuzhin <[email protected]>

azat · 2022-01-27T17:09:49Z

alexey-milovidov edited yesterday

Changelog category (leave one):
-Improvement
+Bug Fix (user-visible misbehaviour in official stable or prestable release)

@alexey-milovidov There were some leaks before, not that significant, but still maybe worth to backport.

azat · 2022-01-27T20:12:50Z

Interesting, more leaks had been found, but there were no leaks in the previous run, will take a look.

I've fixed few leaks, but apparently one leak had been left, it is hard to trigger, it is due to some exception safety in Aggregator code... Will take a look later.

alexey-milovidov · 2022-01-29T22:06:43Z

It does not work at all:

ubuntu@ip-172-31-13-60:~$ curl https://clickhouse.com/ | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1234    0  1234    0     0   6108      0 --:--:-- --:--:-- --:--:--  6108

Will download https://builds.clickhouse.com/master/amd64/clickhouse

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1934M  100 1934M    0     0  29.3M      0  0:01:05  0:01:05 --:--:-- 44.7M

Successfully downloaded the ClickHouse binary, you can run it as:
    ./clickhouse

You can also install it:
    sudo ./clickhouse install
ubuntu@ip-172-31-13-60:~$ sudo ./clickhouse install
Copying ClickHouse binary to /usr/bin/clickhouse.new
Renaming /usr/bin/clickhouse.new to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-server to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-client to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-local to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-benchmark to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-copier to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-obfuscator to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-git-import to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-compressor to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-format to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-extract-from-config to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-keeper to /usr/bin/clickhouse.
Creating symlink /usr/bin/clickhouse-keeper-converter to /usr/bin/clickhouse.
Creating clickhouse group if it does not exist.
 groupadd -r clickhouse
Creating clickhouse user if it does not exist.
 useradd -r --shell /bin/false --home-dir /nonexistent -g clickhouse clickhouse
Will set ulimits for clickhouse user in /etc/security/limits.d/clickhouse.conf.
Creating config directory /etc/clickhouse-server.
Creating config directory /etc/clickhouse-server/config.d that is used for tweaks of main server configuration.
Creating config directory /etc/clickhouse-server/users.d that is used for tweaks of users configuration.
Data path configuration override is saved to file /etc/clickhouse-server/config.d/data-paths.xml.
Log path configuration override is saved to file /etc/clickhouse-server/config.d/logger.xml.
User directory path configuration override is saved to file /etc/clickhouse-server/config.d/user-directories.xml.
OpenSSL path configuration override is saved to file /etc/clickhouse-server/config.d/openssl.xml.
Creating log directory /var/log/clickhouse-server.
Creating data directory /var/lib/clickhouse.
Creating pid directory /var/run/clickhouse-server.
 chown -R clickhouse:clickhouse '/var/log/clickhouse-server'
 chown -R clickhouse:clickhouse '/var/run/clickhouse-server'
 chown  clickhouse:clickhouse '/var/lib/clickhouse'
Enter password for default user: 
Password for default user is empty string. See /etc/clickhouse-server/users.xml and /etc/clickhouse-server/users.d to change it.
Setting capabilities for clickhouse binary. This is optional.
 chown -R clickhouse:clickhouse '/etc/clickhouse-server'

ClickHouse has been successfully installed.

Start clickhouse-server with:
 sudo clickhouse start

Start clickhouse-client with:
 clickhouse-client

ubuntu@ip-172-31-13-60:~$ clickhouse-local 
Cannot read /proc/self/auxv (likely kernel is too old or procfs is not mounted): Too many open files
Aborted (core dumped)

Revert #33957

alexey-milovidov · 2022-01-29T22:42:08Z

@azat maybe better solution is to implement your own setenv?

azat · 2022-01-30T09:42:27Z

@azat maybe better solution is to implement your own setenv?

I'm not sure that this is a good idea, and it will not fix LSan.
Ok, let's try #34152 for now to make LSan work.

It does not work at all:

I guess it is machine in aws.
Which RLIMIT_NOFILE does it have?

alexey-milovidov · 2022-01-30T11:17:26Z

I've already destroyed the machine.
Most likely RLIMIT_NOFILE does not matter.

It works if I use strace.

azat · 2022-01-30T11:25:23Z

Most likely RLIMIT_NOFILE does not matter.

Indeed.

It works if I use strace.

Ok, I guess it can be related to some LSM (yama or similar)

azat · 2022-02-01T19:05:51Z

I've fixed few leaks, but apparently one leak had been left, it is hard to trigger, it is due to some exception safety in Aggregator code... Will take a look later.

Fix memory leak in AggregatingInOrderTransform #34234

getauxval() from glibc-compatibility did not work always correctly: - It does not work after setenv(), and this breaks vsyscalls, like sched_getcpu() [1] (and BaseDaemon.cpp always set TZ if timezone is defined, which is true for CI [2]). Also note, that fixing setenv() will not fix LSan, since the culprit is getauxval() [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1163404 [2]: ClickHouse#32928 (comment) - Another think that is definitely broken is LSan (Leak Sanitizer), it relies on worked getauxval() but it does not work if __environ is not initialized yet (there is even a commit about this). And because of, at least, one leak had been introduced [3]: [3]: ClickHouse#33840 Fix this by using /proc/self/auxv with fallback to environ solution to make it compatible with environment that does not allow reading from auxv (or no procfs). v2: add fallback to environ solution v3: fix return value for __auxv_init_procfs() Refs: ClickHouse#33957 Signed-off-by: Azat Khuzhin <[email protected]>

getauxval() from glibc-compatibility did not work always correctly: - It does not work after setenv(), and this breaks vsyscalls, like sched_getcpu() [1] (and BaseDaemon.cpp always set TZ if timezone is defined, which is true for CI [2]). Also note, that fixing setenv() will not fix LSan, since the culprit is getauxval() [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1163404 [2]: ClickHouse#32928 (comment) - Another think that is definitely broken is LSan (Leak Sanitizer), it relies on worked getauxval() but it does not work if __environ is not initialized yet (there is even a commit about this). And because of, at least, one leak had been introduced [3]: [3]: ClickHouse#33840 Fix this by using /proc/self/auxv with fallback to environ solution to make it compatible with environment that does not allow reading from auxv (or no procfs). v2: add fallback to environ solution v3: fix return value for __auxv_init_procfs() (cherry picked from commit f187c34) v4: more verbose message on errors, CI founds [1]: AUXV already has value (529267711) [1]: https://s3.amazonaws.com/clickhouse-test-reports/39103/2325f7e8442d1672ce5fb43b11039b6a8937e298/stress_test__memory__actions_.html v5: break at AT_NULL v6: ignore AT_IGNORE v7: suppress TSan and remove superior check to avoid abort() in case of race v8: proper suppressions (not inner function but itself) Refs: ClickHouse#33957 Signed-off-by: Azat Khuzhin <[email protected]>

robot-clickhouse added the pr-bugfix Pull request with bugfix, not backported by default label Jan 24, 2022

This was referenced Jan 24, 2022

Revert "Revert "less sys calls #2: make vdso work again"" #28132

Merged

Speed up parts loading process of MergeTree by metadata cache #32928

Merged

alexey-milovidov reviewed Jan 24, 2022

View reviewed changes

base/glibc-compatibility/musl/getauxval.c Outdated Show resolved Hide resolved

azat force-pushed the fix-getauxval branch 2 times, most recently from 8dbfbcb to b8149ad Compare January 25, 2022 15:41

azat changed the title ~~Fix getauxval() in glibc-compatibility~~ Fix getauxval() in glibc-compatibility and fix LSan reports Jan 25, 2022

azat changed the title ~~Fix getauxval() in glibc-compatibility and fix LSan reports~~ Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work) Jan 25, 2022

azat force-pushed the fix-getauxval branch from b8149ad to e244c6c Compare January 25, 2022 15:42

alexey-milovidov self-assigned this Jan 26, 2022

alexey-milovidov reviewed Jan 26, 2022

View reviewed changes

base/glibc-compatibility/musl/getauxval.c Outdated Show resolved Hide resolved

alexey-milovidov reviewed Jan 26, 2022

View reviewed changes

base/glibc-compatibility/musl/getauxval.c Outdated Show resolved Hide resolved

alexey-milovidov reviewed Jan 26, 2022

View reviewed changes

base/glibc-compatibility/musl/getauxval.c Outdated Show resolved Hide resolved

azat force-pushed the fix-getauxval branch from e244c6c to 8efa20d Compare January 26, 2022 09:41

azat added 2 commits January 26, 2022 12:56

azat force-pushed the fix-getauxval branch from 8efa20d to 08f4f45 Compare January 26, 2022 09:56

alexey-milovidov approved these changes Jan 26, 2022

View reviewed changes

robot-clickhouse added pr-improvement Pull request with some product improvements and removed pr-bugfix Pull request with bugfix, not backported by default labels Jan 26, 2022

azat marked this pull request as draft January 27, 2022 08:08

azat added 2 commits January 27, 2022 13:46

Add echo for 01570_aggregator_combinator_simple_state

e04b862

Signed-off-by: Azat Khuzhin <[email protected]>

azat marked this pull request as ready for review January 27, 2022 17:07

alexey-milovidov merged commit 63d8d75 into ClickHouse:master Jan 27, 2022

azat deleted the fix-getauxval branch January 28, 2022 04:19

alexey-milovidov added a commit that referenced this pull request Jan 29, 2022

Revert #33957

7135ff4

alexey-milovidov mentioned this pull request Jan 29, 2022

Revert #33957 #34146

Merged

alexey-milovidov added a commit that referenced this pull request Jan 29, 2022

Merge pull request #34146 from ClickHouse/revert-33957

44ebcb9

Revert #33957

azat mentioned this pull request Jan 30, 2022

Fix LSan by disabling glibc-compatiblity (due to broken getauxval()) #34152

Closed

alexey-milovidov mentioned this pull request Feb 1, 2022

At least one test without Docker #34241

Closed

azat mentioned this pull request Feb 9, 2022

Fix memory leak in AggregatingInOrderTransform #34234

Merged

azat mentioned this pull request May 16, 2022

Implementation of gwp-asan technique #36826

Closed

azat mentioned this pull request Jul 17, 2022

[RFC] Fix LSan by fixing getauxval() #39299

Merged

azat mentioned this pull request Jul 21, 2022

Fix LSan by fixing getauxval() (resubmit v4) #39430

Merged

Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work) #33957

Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work) #33957

Uh oh!

Conversation

azat commented Jan 24, 2022 • edited by alexey-milovidov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Jan 24, 2022

Uh oh!

alexey-milovidov commented Jan 24, 2022

Uh oh!

azat commented Jan 24, 2022

Uh oh!

alexey-milovidov commented Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented Jan 25, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Jan 26, 2022

Uh oh!

azat commented Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented Jan 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat commented Jan 27, 2022

Uh oh!

alexey-milovidov commented Jan 29, 2022

Uh oh!

alexey-milovidov commented Jan 29, 2022

Uh oh!

azat commented Jan 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Jan 30, 2022

Uh oh!

azat commented Jan 30, 2022

Uh oh!

azat commented Feb 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

azat commented Jan 24, 2022 •

edited by alexey-milovidov

Loading

alexey-milovidov commented Jan 24, 2022 •

edited

Loading

azat commented Jan 27, 2022 •

edited

Loading

azat commented Jan 27, 2022 •

edited

Loading

azat commented Jan 30, 2022 •

edited

Loading