Security Server Ubuntu 18.04 to 20.04 In-place Upgrade.
This document describes the steps required for upgrading a stand-alone Security Server host from Ubuntu 18.04 LTS host to Ubuntu 20.04 LTS. Please
read carefully through the whole document before starting the upgrade process. It is assumed that the reader is familiar with the Ubuntu Linux distribution
and has experience of Ubuntu release upgrades.
With the upgrade to Ubuntu 20.04 LTS, PostgreSQL is updated from version 10 to 12.
Alternative method: Upgrading Security Server to Ubuntu 20.04 Using a Configuration Backup
Preparations
If upgrading a system that uses a hardware security module: Please verify that the HSM is compatible with Ubuntu 20.04 and check the HSM module
documentation for upgrade instructions.
Review the Ubuntu release notes and upgrade instructions:
18.04 20.04 https://wiki.ubuntu.com/FocalFossa/ReleaseNotes
Update all packages to the latest versions:
apt update && apt full-upgrade
Ensure that the X-Road software version is at least 6.25.0
Recommended: Make sure that you have an up-to-date backup (or virtual machine snapshot) of the server and database.
Use the admin UI to take a backup of the security server configuration and download it to a safe location
This makes it possible to restore the server configuration if the upgrade fails for some reason.
Stop the security server and prevent it from starting automatically at boot
sudo systemctl stop "xroad-*"
sudo systemctl disable "xroad-*"
Optionally, take backup of the message log database. Without a backup, a failure in the upgrade process may cause loss of data.
See https://www.postgresql.org/docs/10/backup.html
Using pg_dump is a straightforward option, but can be problematic if the database is extremely large:
sudo -iu postgres pg_dump -d messagelog -F c -f <dump_file>
Reducing the size of the message log database dump: By default the message log database keeps 30 days of message records but one
can (temporarily) change the retention time in order to reduce the number of records in the database, thus reducing the size of the
database dump. Changing the setting has no immediate effect since by default the message log cleanup is run twice a day (noon and
midnight). See Security Server User Guide for more information.
Optionally, take a backup of the archived message log files.
Note that some of the logged messages might not have been archived yet and exist only in the database.
Optionally, take backup of the operational monitoring database (if xroad-opmonitoring is installed). Without a backup, a failure in the upgrade
process may cause loss of data.
See https://www.postgresql.org/docs/10/backup.html
sudo -iu postgres pg_dump -d "op-monitor" -F c -f <dump_file>
Upgrading Ubuntu 18.04 to 20.04
Install update-manager-core if it is not already installed.
Make sure the Prompt line in /etc/update-manager/release-upgrades is set to 'lts'
Launch the upgrade tool with the command sudo do-release-upgrade
Follow the on-screen instructions.
When the upgrade is finished, reboot when prompted.
Upgrading the X-Road software
Upgrade the database
The Ubuntu upgrade process by default creates an empty database instance that should be removed before the old database is upgraded to
version 12.
List the database instances with pg_lsclusters, and drop the extra ones
Do not remove the version 10 (main) running on port 5432
sudo pg_dropcluster --stop <version, e.g. 12> main
Example
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
12 main 5433 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
$ sudo pg_dropcluster --stop 12 main
Upgrade the old database to version 12.
Ubuntu's pg_upgradecluster helper script by default dumps the old database, which can require a lot of time and free disk space if the database is
large. The --method=upgrade uses pg_upgrade instead and the --link option avoids copying the data files.
Using --link requires that the new and old database are in the same filesystem. See https://www.postgresql.org/docs/12/pgupgrade.html for
details and recovery instructions.
sudo pg_upgradecluster --method=upgrade --link 10 main
Example
$ sudo pg_upgradecluster --method=upgrade --link 10 main
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
10 main 5433 down postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
Update the X-Road package repository to point to the Ubuntu 20.04 packages
Update the apt repository:
In the /etc/apt/sources.list.d/xroad.list file, replace the bionic repository with focal repository, so that the file contains:
deb https://x-tee.ee/packages/live/xroad/ focal main
Upgrade the packages
sudo apt update && sudo apt full-upgrade
sudo systemctl enable "xroad-*"
sudo systemctl start "xroad-*"
Verify that the security server services are running and the system is responding
systemctl list-units "xroad-*" "postgresql@*"
UNIT LOAD ACTIVE SUB DESCRIPTION
[email protected] loaded active running PostgreSQL Cluster 12-main
xroad-base.service loaded active exited X-Road initialization
xroad-confclient.service loaded active running X-Road confclient
xroad-monitor.service loaded active running X-Road Monitor*
xroad-opmonitor.service loaded active running X-Road opmonitor daemon*
xroad-proxy.service loaded active running X-Road Proxy
xroad-proxy-ui-api.service loaded active running X-Road Proxy UI REST API
xroad-signer.service loaded active running X-Road signer
* if the monitoring addon(s) are installed
Drop old database and obsolete packages
After verifying that the database upgrade was successful, drop the old database and remove obsolete PostgreSQL packages.
sudo pg_dropcluster 10 main
sudo apt purge postgresql-10
sudo apt autoremove
Troubleshooting
If recovery from a upgrade failure is not possible and there is a backup/snapshot available, one can restore it and start over.
If all else fails, do a clean Ubuntu 20.04 installation and restore the security server configuration from the backup.
