postgres_data/

Copyright © `2024` `Chandan <[email protected]>`

Permission is hereby granted, free of charge, to any person

obtaining a copy of this software and associated documentation

files (the “Software”), to deal in the Software without

restriction, including without limitation the rights to use,

copy, modify, merge, publish, distribute, sublicense, and/or sell

copies of the Software, and to permit persons to whom the

Software is furnished to do so, subject to the following

conditions:

The above copyright notice and this permission notice shall be

included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND,

EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES

OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT

HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,

WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING

FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR

OTHER DEALINGS IN THE SOFTWARE.

I'm making my Postgres setup open source. This is what I use for self-hosting PostgreSQL myself.

It comes with a lot of room for customization and notably:

- Automatic SSL certificate generation/renewals with Traefik as a reverse proxy

- PgBouncer as a connection pooler that uses auth query instead of userlist.txt

- Automatic incremental backups to S3-compatible storage

- Script to create databases and users with granular, scoped permissions within the single cluster

I would like to hear your thoughts, suggestions, and recommendations regarding this setup, and if anything can be improved.

I want to keep this setup small and without bloatware. It can be used as is, but I expect it to be customized according to your needs.

Current config files expect 4 GB of RAM. If you have less or more, change the settings in the postgres.conf file and the Docker Compose service memory limits.

Currently, tools to view Postgres are not added as part of the setup.

You can access the database through psql or tools like [pgAdmin 4](https://github.com/pgadmin-org/pgadmin4), [Beekeeper Studio](https://github.com/beekeeper-studio/beekeeper-studio), or [DBeaver](https://github.com/dbeaver/dbeaver).

**Assumptions:** You have a Linux server used solely for hosting PostgreSQL with Docker.

1. `git clone https://github.com/realchandan/postgres_setup.git`

2. `cd postgres_setup`

3. Copy environment files:

```bash

cp .env.example .env

cp ./config/postgres.env.example ./config/postgres.env

cp ./config/pgbackup.env.example ./config/pgbackup.env

```

Then, modify the environment files with the appropriate values.

Here's an explanation of environment variables:

> **.env**

> | Variable Name | Explanation |

> | --------------- | ---------------------------------------------------------------------------------- |

> | ACME_EMAIL | The email to be used for ACME/LetsEncrypt |

> | POSTGRES_DOMAIN | The domain where you want to host the database over SSL, e.g. postgres.example.com |

> **./config/postgres.env**

> | Variable Name | Explanation |

> | ----------------- | ------------------------------------------------------------------------------------------------- |

> | POSTGRES_DB | The name of the default database. Ideally, you shouldn’t change it (by default, it's postgres). |

> | POSTGRES_PASSWORD | The password of the PostgreSQL superuser (set a very strong one here). |

> | POSTGRES_USER | The username of the superuser (ideally, don’t change it). |

> **./config/pgbackup.env**

> Refer [here](https://github.com/realchandan/pgbackup?tab=readme-ov-file#usage). If you don't want backups, comment out the pgbackup service in the Docker Compose file.

4. Point your domain A/AAAA records to the server’s public IPv4/IPv6 addresses.

5. Allow ports 443 and 5432 (TCP) through the firewall. Depending on your firewall, steps may vary. Port 443 is needed for Let’s Encrypt TLS challenge, and 5432 is used by PgBouncer.

6. Add public permissions to the `./config/pg` folder with `chmod -R 777 ./config/pg`.

7. Run `docker compose --env-file .env up -d` to bring up all the services.

8. Create a new database using:

```bash

docker exec -it postgres bash -c "/docker-entrypoint-initdb.d/create-user.sh awesome_db passw0rd"

```

This command creates a user called `awesome_db_user` with the password `passw0rd` and gives them access to a database named `awesome_db`.

9. Enjoy and star this repo! (Helps me flex!)

local all all trust

host all pgbouncer 10.10.10.105/32 trust

host all pgbouncer fd00:bbbb:cccc:dddd::105/128 trust

host all all all md5

host replication postgres all md5

listen_addresses = '*'

max_connections = 30

password_encryption = md5

port = 5432

# We got 4 GB RAM for postgres

autovacuum_max_workers = 3

effective_cache_size = 2GB # between 50% and 75% of the memory

effective_io_concurrency = 180 # for NVME SSD

log_min_duration_statement = 1000ms

maintenance_work_mem = 64MB

max_prepared_transactions = 30 # same as max_connections

random_page_cost =1

seq_page_cost = 1

shared_buffers = 1GB # 25% of the memory

work_mem = 5MB

hba_file = '/etc/conf/pg_hba.conf'

summarize_wal=on

checkpoint_timeout = 5min

max_wal_size = 1GB

log_checkpoints = 'on'

set -e

password=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c 64)

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE USER pgbouncer WITH ENCRYPTED PASSWORD '$password';"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "

set -e

function create_database() {

local database="$1"

local password="$2"

local username="${database}_user"

local role="${database}_role"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE ROLE $role NOLOGIN;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE DATABASE $database OWNER $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "REVOKE ALL ON SCHEMA public FROM PUBLIC;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "REVOKE ALL ON DATABASE $database FROM PUBLIC;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT CONNECT ON DATABASE $database TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT ALL PRIVILEGES ON DATABASE $database TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT ALL PRIVILEGES ON SCHEMA public TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON TABLES TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON SEQUENCES TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$database" -c "ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON FUNCTIONS TO $role;"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "CREATE USER $username WITH ENCRYPTED PASSWORD '$password';"

psql -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -c "GRANT $role to $username;"

}

if [[ -z "$1" || -z "$2" ]]; then

echo "Usage: $0 <database_name> <password>"

exit 0

create_database "$1" "$2"

DB_HOST=postgres

DB_PORT=5432

REMOTE_FOLDER=my_db_backups

S3_ACCESS_KEY=

S3_BUCKET_NAME=

S3_ENDPOINT=

S3_REGION=

S3_SECRET_KEY=

SCHEDULE=*/5 * * * *

*=host=postgres port=5432

listen_addr = *

listen_port = 6432

auth_type = md5

default_pool_size = 15

max_client_conn = 1500

pool_mode = transaction

server_idle_timeout = 600

server_lifetime = 3600

auth_user=pgbouncer

auth_query = SELECT usename, passwd FROM user_search($1)

auth_dbname=postgres

max_prepared_statements = 100