Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Keep profile and history data in shared memory #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

Keep profile and history data in shared memory #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
49ff16c
Keep profile and history data in shared memory
Dec 27, 2022
ef33592
Resolve clang warning regarding to variable initialization
Feb 1, 2023
3bfb23d
Fix estimated size for profile hash table
maksm90 Feb 10, 2023
08b7e8c
Refactoring: s/pgws_hash/pgws_profile_hash/ s/pgws_hash_lock/pgws_pr…
maksm90 Feb 10, 2023
f0b62aa
Refine FIXME in collector's loop over ProcArray
maksm90 Feb 10, 2023
9a69965
Add TODO item to reduce contention of ProcArrayLock
maksm90 Feb 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 35 additions & 23 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,27 @@ When `pg_wait_sampling` is enabled, it collects two kinds of statistics.
recent samples depending on history size (configurable). Assuming there is
a client who periodically read this history and dump it somewhere, user
can have continuous history.
* Waits profile. It's implemented as in-memory hash table where count
of samples are accumulated per each process and each wait event
(and each query with `pg_stat_statements`). This hash
table can be reset by user request. Assuming there is a client who
periodically dumps profile and resets it, user can have statistics of
intensivity of wait events among time.

In combination with `pg_stat_statements` this extension can also provide
per query statistics.
* Waits profile. It's implemented as bounded in-memory hash table where counts
of samples are accumulated per triple of process pid, wait event and query id
(when its computing is enabled on PG server, on versions below 14 this
requires `pg_stat_statements` extension). The least used entries are evicted
when overflow of hash table is encountered. Hash table also can be reset by
user request. Assuming there is a client who periodically dumps profile and
computes differential counters from adjacent dumps, user can have statistics
of intensivity of wait events among time.

Starting from PG14 this extension might activate computing of query id on server
side to enable per query id statistics. The older PG versions require to install
`pg_stat_statements` extension for this purpose.

`pg_wait_sampling` launches special background worker for gathering the
statistics above.

The profile statistics as well as history items are not persisted to disk so
server restart resets all already accummulated data. This is not crucial for
profile counters because we are primarily interested in differential values, not
absolute values of these counters.

Availability
------------

Expand Down Expand Up @@ -125,24 +133,28 @@ in-memory hash table.
The work of wait event statistics collector worker is controlled by following
GUCs.

| Parameter name | Data type | Description | Default value |
| ----------------------------------- | --------- | ------------------------------------------- | ------------: |
| pg_wait_sampling.history_size | int4 | Size of history in-memory ring buffer | 5000 |
| pg_wait_sampling.history_period | int4 | Period for history sampling in milliseconds | 10 |
| pg_wait_sampling.profile_period | int4 | Period for profile sampling in milliseconds | 10 |
| pg_wait_sampling.profile_pid | bool | Whether profile should be per pid | true |
| pg_wait_sampling.profile_queries | bool | Whether profile should be per query | true |
| Parameter name | Data type | Description | Default value | Change policy |
| ------------------------------------ | --------- | ----------------------------------------------------------------------------------- | ------------- | ------------- |
| pg_wait_sampling.max_profile_entries | int4 | Maximum number of entries in profile hash table | 5000 | restart |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just profile_size, like history_size?

Copy link
Collaborator Author

@maksm90 maksm90 Feb 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly mimicked to max_connections name, the prefix _size for me is associated to something measured in bytes

| pg_wait_sampling.history_size | int4 | Size of history in-memory ring buffer | 5000 | restart |
| pg_wait_sampling.profile_period | int4 | Period for profile sampling in milliseconds (zero value disables profile gathering) | 10 | reload |
| pg_wait_sampling.history_period | int4 | Period for history sampling in milliseconds (zero value disables history gathering) | 0 | reload |
| pg_wait_sampling.profile_pid | bool | Whether profile should be per pid | true | restart |
| pg_wait_sampling.profile_queries | bool | Whether profile should be per query | true | restart |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does changing profile_pid or profile_queries really require a restart?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now it looks redundantly. This is intended for future work.
Tracking pid would define whether we have to add a separate hash table to account counters for currently active processes when we have another hash table for historic data that accumulates counters for finished ones. Or how big a single hash table for historic and actual statistics have to be when statistics from completed processes is flushed to corresponding historic entries.
Similarly for tracking queryid: taking into account that the number of pid values and wait_events is bounded, we might more finely tune required memory for profile hash table by knowing about queryid tracking and expected maximum number of tracked queryid values (or another future fields with variable quantity in profile key) when this option is on.

In addition, it's some confusing to have entries in profile with null values and not null ones in pid and/or queryid fields corresponding in fact to the same entries. This requires profile reset after setting applying.


If `pg_wait_sampling.profile_pid` is set to false, sampling profile wouldn't be
collected in per-process manner. In this case the value of pid could would
be always zero and corresponding row contain samples among all the processes.
collected in per-process manner. In this case the value of pid will be NULL and
corresponding rows contain samples among all the processes.

While `pg_wait_sampling.profile_queries` is set to false `queryid` field in
views will be zero.
__Caution__:
When sampling per pid is enabled, all profile entries for already completed
processes are left in hash table. Therefore, it's neccessary to take into
account periodic flushing of profile to prevent recycling of 32-bit pid values
in profile hash table and as consequence possible increments to profile entries
belonging to some old processes with the same pid values as for current ones.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think pid recycling is such an important concern in practice to justify this warning? Wouldn't we run out of hash table entries and evict the old ones, before we hit a recycled pid. I'm not sure that this is frequent enough to draw user's attention to.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, theoretically confusion caused by pid recycling is possible. And we have to frankly declare about it, I suppose

Copy link
Collaborator Author

@maksm90 maksm90 Feb 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, it's wise to notice in doc that if the size of profile hash table is reasonably not huge and there are no other than postgres actively spawn processes in a system the probability of such pid recycling goes to zero


These GUCs are allowed to be changed by superuser. Also, they are placed into
shared memory. Thus, they could be changed from any backend and affects worker
runtime.
While `pg_wait_sampling.profile_queries` is set to false `queryid` field in
views will be NULL.

See
[PostgreSQL documentation](http://www.postgresql.org/docs/devel/static/monitoring-stats.html#WAIT-EVENT-TABLE)
Expand Down
Loading