Thanks to visit codestin.com
Credit goes to github.com

Skip to content

DEVELOPERS.md

Latest commit

 

History

History
569 lines (373 loc) · 15.9 KB

DEVELOPERS.md

File metadata and controls

569 lines (373 loc) · 15.9 KB

Developer guide

Install PostgreSQL

For RPM based distributions such as Fedora and RHEL you can add the PostgreSQL YUM repository and do the install via

Fedora 42

rpm -Uvh https://download.postgresql.org/pub/repos/yum/reporpms/F-42-x86_64/pgdg-redhat-repo-latest.noarch.rpm

RHEL 9.x / Rocky Linux 9.x

x86_64

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
rpm -Uvh https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
dnf config-manager --set-enabled crb

aarch64

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
rpm -Uvh https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-aarch64/pgdg-redhat-repo-latest.noarch.rpm
dnf config-manager --set-enabled crb

PostgreSQL 17

dnf -qy module disable postgresql
dnf install -y postgresql17 postgresql17-server postgresql17-contrib postgresql17-libs

This will install PostgreSQL 17.

Install pgmoneta

Pre-install

Basic dependencies

dnf install git gcc clang clang-analyzer clang-tools-extra cmake make libev libev-devel openssl openssl-devel systemd systemd-devel zlib zlib-devel libzstd libzstd-devel lz4 lz4-devel libssh libssh-devel python3-docutils libatomic bzip2 bzip2-devel libarchive libarchive-devel libasan libasan-static

Generate user and developer guide

This process is optional by using the cmake variable -DDOCS=TRUE (default), or -DDOCS=FALSE. If you choose not to generate the PDF and HTML files, you can opt out of downloading these dependencies, and the process will automatically skip the generation.

  1. Download dependencies

    dnf install pandoc texlive-scheme-basic

  2. Download Eisvogel

    Use the command pandoc --version to locate the user data directory. On Fedora systems, this directory is typically located at $HOME/.local/share/pandoc.

    Download the Eisvogel template for pandoc, please visit the pandoc-latex-template repository. For a standard installation, you can follow the steps outlined below.

    wget https://github.com/Wandmalfarbe/pandoc-latex-template/releases/download/v3.4.0/Eisvogel-3.4.0.tar.gz
    tar -xzf Eisvogel-3.4.0.tar.gz
    mkdir -p $HOME/.local/share/pandoc/templates
    mv Eisvogel-3.4.0/eisvogel.latex $HOME/.local/share/pandoc/templates/

  1. Add package for LaTeX

    Download the additional packages required for generating PDF and HTML files.

    dnf install 'tex(fvextra.sty)' 'tex(footnote.sty)' 'tex(footnotebackref.sty)' 'tex(pagecolor.sty)' 'tex(hardwrap.sty)' 'tex(mdframed.sty)' 'tex(sourcesanspro.sty)' 'tex(ly1enc.def)' 'tex(sourcecodepro.sty)' 'tex(titling.sty)' 'tex(csquotes.sty)' 'tex(zref-abspage.sty)' 'tex(needspace.sty)' 'tex(selnolig.sty)' texlive-collection-latexextra

Generate API guide

This process is optional by using the cmake variable -DDOCS=TRUE (default), or -DDOCS=FALSE. If you choose not to generate the API HTML files, you can opt out of downloading these dependencies, and the process will automatically skip the generation.

Download dependencies

    dnf install graphviz doxygen

Generating Code Coverage

Code coverage is automatically enabled only for GCC builds if both gcov and gcovr are installed on your system. To install the required tools, run:

dnf install gcovr gcc

Note: In many distributions, gcovr may not be available as a DNF package. In such cases, you can install it using pip:

pip3 install gcovr

When these tools are present and the compiler is set to GCC, the build system will detect them and enable code coverage generation automatically during the build process. If you use Clang as the compiler, code coverage will not be enabled by default.

Build

cd /usr/local
git clone https://github.com/pgmoneta/pgmoneta.git
cd pgmoneta
mkdir build
cd build
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_BUILD_TYPE=Debug ..
make
make install

This will install pgmoneta in the /usr/local hierarchy with the debug profile.

Check version

You can navigate to build/src and execute ./pgmoneta -? to make the call. Alternatively, you can install it into /usr/local/ and call it directly using:

pgmoneta -?

If you see an error saying error while loading shared libraries: libpgmoneta.so.0: cannot open shared object running the above command. you may need to locate where your libpgmoneta.so.0 is. It could be in /usr/local/lib or /usr/local/lib64 depending on your environment. Add the corresponding directory into /etc/ld.so.conf.

To enable these directories, you would typically add the following lines in your /etc/ld.so.conf file:

/usr/local/lib
/usr/local/lib64

Remember to run ldconfig to make the change effective.

Setup pgmoneta

Let's give it a try. The basic idea here is that we will use two users: one is postgres, which will run PostgreSQL, and one is pgmoneta, which will run pgmoneta to do backup of PostgreSQL.

In many installations, there is already an operating system user named postgres that is used to run the PostgreSQL server. You can use the command

getent passwd | grep postgres

to check if your OS has a user named postgres. If not use

useradd -ms /bin/bash postgres
passwd postgres

If the postgres user already exists, don't forget to set its password for convenience.

1. postgres

Open a new window, switch to the postgres user. This section will always operate within this user space.

sudo su -
su - postgres

Initialize cluster

If you use dnf to install your postgresql, chances are the binary file is in /usr/bin/

export PATH=/usr/bin:$PATH
initdb -k /tmp/pgsql

Remove default access

Remove last lines from /tmp/pgsql/pg_hba.conf

host    all             all             127.0.0.1/32            trust
host    all             all             ::1/128                 trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

Add access for users and a database

Add new lines to /tmp/pgsql/pg_hba.conf

host    mydb             myuser          127.0.0.1/32            scram-sha-256
host    mydb             myuser          ::1/128                 scram-sha-256
host    postgres         repl            127.0.0.1/32            scram-sha-256
host    postgres         repl            ::1/128                 scram-sha-256
host    replication      repl            127.0.0.1/32            scram-sha-256
host    replication      repl            ::1/128                 scram-sha-256

Set password_encryption

Set password_encryption value in /tmp/pgsql/postgresql.conf to be scram-sha-256

password_encryption = scram-sha-256

For version 13, the default is md5, while for version 14 and above, it is scram-sha-256. Therefore, you should ensure that the value in /tmp/pgsql/postgresql.conf matches the value in /tmp/pgsql/pg_hba.conf.

Set replication level

Set wal_level value in /tmp/pgsql/postgresql.conf to be replica

wal_level = replica

Set summarize_wal

Set summarize_wal value in /tmp/pgsql/postgresql.conf to be on in order to have Online: true in pgmoneta server status.

i.e This is required only for PostgreSQL version 17 and above.

summarize_wal = on

Start PostgreSQL

pg_ctl  -D /tmp/pgsql/ start

Here, you may encounter issues such as the port being occupied or permission being denied. If you experience a failure, you can go to /tmp/pgsql/log to check the reason.

You can use

pg_isready

to test

Add users and a database

export PATH=/usr/pgsql-17/bin:$PATH
createuser -P myuser
createdb -E UTF8 -O myuser mydb

Then

psql postgres
CREATE ROLE repl WITH LOGIN REPLICATION PASSWORD 'secretpassword';
\q

Add replication slot

Add the required replication slot

psql postgres
SELECT pg_create_physical_replication_slot('repl', true, false);
\q

Alternatively, configure automatically slot creation by adding create_slot = yes to [pgmoneta] or corresponding server section.

Verify access

For the user myuser (standard) use mypass

psql -h localhost -p 5432 -U myuser mydb
\q

For the user repl (pgmoneta) use secretpassword

psql -h localhost -p 5432 -U repl postgres
\q

Add pgmoneta user

sudo su -
useradd -ms /bin/bash pgmoneta
passwd pgmoneta
exit

2. pgmoneta

Open a new window, switch to the pgmoneta user. This section will always operate within this user space.

sudo su -
su - pgmoneta

Create base directory

mkdir backup

Create pgmoneta configuration

Add the master key

pgmoneta-admin master-key

You have to choose a password for the master key and it must be at least 8 characters - remember it!

then create vault

pgmoneta-admin -f pgmoneta_users.conf -U repl -P secretpassword user add

Input the replication user and its password to grant pgmoneta access to the database. Ensure that the information is correct.

Create the pgmoneta.conf configuration file to use when running pgmoneta.

cat > pgmoneta.conf
[pgmoneta]
host = *
metrics = 5001

base_dir = /home/pgmoneta/backup

compression = zstd

retention = 7

log_type = file
log_level = info
log_path = /tmp/pgmoneta.log

unix_socket_dir = /tmp/

[primary]
host = localhost
port = 5432
user = repl
wal_slot = repl

And create the pgmoneta_cli.conf configuration file to use when running pgmoneta-cli.

cat > pgmoneta_cli.conf
[pgmoneta-cli]
compression = zstd

log_type = file
log_level = info
log_path = /tmp/pgmoneta-cli.log

unix_socket_dir = /tmp/

In our main section called [pgmoneta] we setup pgmoneta to listen on all network addresses. We will enable Prometheus metrics on port 5001 and have the backups live in the /home/pgmoneta/backup directory. All backups are being compressed with zstd and kept for 7 days. Logging will be performed at info level and put in a file called /tmp/pgmoneta.log. Last we specify the location of the unix_socket_dir used for management operations.

Next we create a section called [primary] which has the information about our PostgreSQL instance. In this case it is running on localhost on port 5432 and we will use the repl user account to connect.

Finally, you should be able to obtain the version of pgmoneta. Cheers!

Start pgmoneta

pgmoneta -c pgmoneta.conf -u pgmoneta_users.conf

Create a backup

open a new terminal and log in with pgmoneta

pgmoneta-cli -c pgmoneta_cli.conf backup primary

View backup

pgmoneta-cli -c pgmoneta_cli.conf status details

Shutdown pgmoneta

pgmoneta-cli -c pgmoneta_cli.conf shutdown

WAL Tools

pgmoneta provides two WAL (Write-Ahead Log) tools for working with PostgreSQL WAL files:

  • pgmoneta-walinfo: Read and display information about WAL files
  • pgmoneta-walfilter: Filter WAL files based on user-defined rules

For detailed user documentation about these tools, please refer to the WAL Tools chapter.

For developer information about the internal APIs and implementation details, see the WAL developer guide.

XID64

pgmoneta is compatible with external PostgreSQL patches that introduce 64-bit Transaction IDs (FullTransactionId).

Support includes:

  • Backup/Restore: Correct handling of 64-bit XIDs during backup and restore operations.

Note: WAL tools are not supported with XID64.

Logging levels

Level Description
TRACE Information for developers including values of variables
DEBUG Higher level information for developers - typically about flow control and the value of key variables
INFO A user command was successful or general health information about the system
WARN A user command didn't complete correctly so attention is needed
ERROR Something unexpected happened - try to give information to help identify the problem
FATAL We can't recover - display as much information as we can about the problem and exit(1)

End

Now that we've attempted our first backup, take a moment to relax. There are a few things we need to pay attention to:

  1. Since we initialized the database in /tmp, the data in this directory might be removed after you go offline, depending on your OS configuration. If you want to make it permanent, choose a different directory.

  2. Always use clang-format to format your code when you make modifications.

Testing

See doc/TEST.md for adding test cases and running test suites. It is recommended that you ALWAYS run tests before raising PR.

C programming

pgmoneta is developed using the C programming language so it is a good idea to have some knowledge about the language before you begin to make changes.

There are books like,

that can help you

Debugging

In order to debug problems in your code you can use gdb, or add extra logging using the pgmoneta_log_XYZ() API

Core APIs

You may find core APIs quite useful. Try not to reinvent the wheels, unless for a good reason.

Basic git guide

Here are some links that will help you

Start by forking the repository

This is done by the "Fork" button on GitHub.

Clone your repository locally

This is done by

git clone [email protected]:<username>/pgmoneta.git

Add upstream

Do

cd pgmoneta
git remote add upstream https://github.com/pgmoneta/pgmoneta.git

Do a work branch

git checkout -b mywork main

Make the changes

Remember to verify the compile and execution of the code

AUTHORS

Remember to add your name to the following files,

AUTHORS
doc/manual/en/97-acknowledgement.md

in your first pull request

Multiple commits

If you have multiple commits on your branch then squash them

git rebase -i HEAD~2

for example. It is p for the first one, then s for the rest

Rebase

Always rebase

git fetch upstream
git rebase -i upstream/main

Force push

When you are done with your changes force push your branch

git push -f origin mywork

and then create a pull requests for it

Repeat

Based on feedback keep making changes, squashing, rebasing and force pushing

PTAL

When you are working on a change put it into Draft mode, so we know that you are not happy with it yet.

Please, send a PTAL to the Committer that were assigned to you once you think that your change is complete. And, of course, take it out of Draft mode.

Undo

Normally you can reset to an earlier commit using git reset <commit hash> --hard. But if you accidentally squashed two or more commits, and you want to undo that, you need to know where to reset to, and the commit seems to have lost after you rebased.

But they are not actually lost - using git reflog, you can find every commit the HEAD pointer has ever pointed to. Find the commit you want to reset to, and do git reset --hard.