Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 4783e61

Browse files
gsmolkgsmolk
committed
Merge branch 'master' into issue_103
2 parents 759aefc + 535e417 commit 4783e61

File tree

5 files changed

+220
-21
lines changed

5 files changed

+220
-21
lines changed

‎Documentation.md

+18-4
Original file line numberDiff line numberDiff line change
@@ -453,12 +453,16 @@ Redardless of data checksums been enabled or not, pg_probackup always check page
453453

454454
#### External directories
455455

456-
To back up a directory located outside of the data directory, use the optional `--external-dirs` parameter that specifies the path to this directory. If you would like to add more than one external directory, provide several paths separated by colons.
456+
To back up a directory located outside of the data directory, use the optional `--external-dirs` parameter that specifies the path to this directory. If you would like to add more than one external directory, provide several paths separated by colons, on Windows system paths must be separated by semicolon instead.
457457

458-
For example, to include '/etc/dir1/' and '/etc/dir2/' directories into the full backup of your *instance_name* instance that will be stored under the *backup_dir* directory, run:
458+
For example, to include `'/etc/dir1/'` and `'/etc/dir2/'` directories into the full backup of your *instance_name* instance that will be stored under the *backup_dir* directory, run:
459459

460460
pg_probackup backup -B backup_dir --instance instance_name -b FULL --external-dirs=/etc/dir1:/etc/dir2
461461

462+
For example, to include `'C:\dir1\'` and `'C:\dir2\'` directories into the full backup of your *instance_name* instance that will be stored under the *backup_dir* directory on Windows system, run:
463+
464+
pg_probackup backup -B backup_dir --instance instance_name -b FULL --external-dirs=C:\dir1;C:\dir2
465+
462466
pg_probackup creates a separate subdirectory in the backup directory for each external directory. Since external directories included into different backups do not have to be the same, when you are restoring the cluster from an incremental backup, only those directories that belong to this particular backup will be restored. Any external directories stored in the previous backups will be ignored.
463467

464468
To include the same directories into each backup of your instance, you can specify them in the pg_probackup.conf configuration file using the [set-config](#set-config) command with the `--external-dirs` option.
@@ -582,6 +586,14 @@ The typical workflow is as follows:
582586

583587
- Run [backup](#backup) or [restore](#restore) commands with [remote options](#remote-mode-options) **on backup host**. pg_probackup connects to the remote system via SSH and creates a backup locally or restores the previously taken backup on the remote system, respectively.
584588

589+
For example, to create archive full backup using remote mode through SSH connection to user `postgres` on host with address `192.168.0.2` via port `2302`, run:
590+
591+
pg_probackup backup -B backup_dir --instance instance_name -b FULL --remote-user=postgres --remote-host=192.168.0.2 --remote-port=2302
592+
593+
For example, to restore latest backup on remote system using remote mode through SSH connection to user `postgres` on host with address `192.168.0.2` via port `2302`, run:
594+
595+
pg_probackup restore -B backup_dir --instance instance_name --remote-user=postgres --remote-host=192.168.0.2 --remote-port=2302
596+
585597
>NOTE: The remote backup mode is currently unavailable for Windows systems.
586598
587599
### Running pg_probackup on Parallel Threads
@@ -953,8 +965,10 @@ Specifies the backup mode to use. Possible values are:
953965
- PAGE — creates an incremental PAGE backup based on the WAL files that have changed since the previous full or incremental backup was taken.
954966
- PTRACK — creates an incremental PTRACK backup tracking page changes on the fly.
955967

956-
-C
957-
--smooth-checkpoint
968+
```
969+
-C
970+
--smooth-checkpoint
971+
```
958972
Spreads out the checkpoint over a period of time. By default, pg_probackup tries to complete the checkpoint as soon as possible.
959973

960974
--stream

‎README.md

+7-7
Original file line numberDiff line numberDiff line change
@@ -58,15 +58,15 @@ Regardless of the chosen backup type, all backups taken with `pg_probackup` supp
5858
### Linux Installation
5959
```shell
6060
#DEB Ubuntu|Debian Packages
61-
echo "deb [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" > /etc/apt/sources.list.d/pg_probackup.list
62-
wget -O - http://repo.postgrespro.ru/pg_probackup/keys/GPG-KEY-PG_PROBACKUP | apt-key add - && apt-get update
63-
apt-get install pg-probackup-{11,10,9.6,9.5}
64-
apt-get install pg-probackup-{11,10,9.6,9.5}-dbg
61+
sudo echo "deb [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" > /etc/apt/sources.list.d/pg_probackup.list
62+
sudo wget -O - http://repo.postgrespro.ru/pg_probackup/keys/GPG-KEY-PG_PROBACKUP | sudo apt-key add - && sudo apt-get update
63+
sudo apt-get install pg-probackup-{11,10,9.6,9.5}
64+
sudo apt-get install pg-probackup-{11,10,9.6,9.5}-dbg
6565

6666
#DEB-SRC Packages
67-
echo "deb-src [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" >>\
67+
sudo echo "deb-src [arch=amd64] http://repo.postgrespro.ru/pg_probackup/deb/ $(lsb_release -cs) main-$(lsb_release -cs)" >>\
6868
/etc/apt/sources.list.d/pg_probackup.list
69-
apt-get source pg-probackup-{11,10,9.6,9.5}
69+
sudo apt-get source pg-probackup-{11,10,9.6,9.5}
7070

7171
#RPM Centos Packages
7272
rpm -ivh http://repo.postgrespro.ru/pg_probackup/keys/pg_probackup-repo-centos.noarch.rpm
@@ -112,7 +112,7 @@ gen_probackup_project.pl C:\path_to_postgresql_source_tree
112112

113113
## Documentation
114114

115-
Currently the latest documentation can be found at [github](https://postgrespro.github.io/pg_probackup)
115+
Currently the latest documentation can be found at [github](https://postgrespro.github.io/pg_probackup).
116116
Slightly outdated documentation can be found at [Postgres Pro Enterprise documentation](https://postgrespro.com/docs/postgrespro/current/app-pgprobackup).
117117

118118
## License

‎src/backup.c

-1
Original file line numberDiff line numberDiff line change
@@ -2493,7 +2493,6 @@ StreamLog(void *arg)
24932493
stream_arg->startpos -= stream_arg->startpos % instance_config.xlog_seg_size;
24942494

24952495
/* Initialize timeout */
2496-
stream_stop_timeout = 0;
24972496
stream_stop_begin = 0;
24982497

24992498
#if PG_VERSION_NUM >= 100000

‎src/pg_probackup.c

+5-5
Original file line numberDiff line numberDiff line change
@@ -323,11 +323,11 @@ main(int argc, char *argv[])
323323
|| strcmp(argv[1], "-V") == 0)
324324
{
325325
#ifdef PGPRO_VERSION
326-
fprintf(stderr, "%s %s (Postgres Pro %s %s)\n",
326+
fprintf(stdout, "%s %s (Postgres Pro %s %s)\n",
327327
PROGRAM_NAME, PROGRAM_VERSION,
328328
PGPRO_VERSION, PGPRO_EDITION);
329329
#else
330-
fprintf(stderr, "%s %s (PostgreSQL %s)\n",
330+
fprintf(stdout, "%s %s (PostgreSQL %s)\n",
331331
PROGRAM_NAME, PROGRAM_VERSION, PG_VERSION);
332332
#endif
333333
exit(0);
@@ -638,12 +638,12 @@ main(int argc, char *argv[])
638638
return do_show(current.backup_id);
639639
case DELETE_CMD:
640640
if (delete_expired && backup_id_string)
641-
elog(ERROR, "You cannot specify --delete-expired and --backup-id options together");
641+
elog(ERROR, "You cannot specify --delete-expired and (-i, --backup-id) options together");
642642
if (merge_expired && backup_id_string)
643-
elog(ERROR, "You cannot specify --merge-expired and --backup-id options together");
643+
elog(ERROR, "You cannot specify --merge-expired and (-i, --backup-id) options together");
644644
if (!delete_expired && !merge_expired && !delete_wal && !backup_id_string)
645645
elog(ERROR, "You must specify at least one of the delete options: "
646-
"--expired |--wal |--merge-expired |--delete-invalid |--backup_id");
646+
"--delete-expired |--delete-wal |--merge-expired |(-i, --backup-id)");
647647
if (!backup_id_string)
648648
return do_retention();
649649
else

‎tests/retention.py

+190-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import os
22
import unittest
33
from datetime import datetime, timedelta
4-
from .helpers.ptrack_helpers import ProbackupTest
4+
from .helpers.ptrack_helpers import ProbackupTest, ProbackupException
55
from time import sleep
66

77

@@ -1264,16 +1264,156 @@ def test_window_error_backups(self):
12641264
self.set_archiving(backup_dir, 'node', node)
12651265
node.slow_start()
12661266

1267-
node.pgbench_init(scale=3)
1268-
12691267
# Take FULL BACKUPs
12701268
backup_id_a1 = self.backup_node(backup_dir, 'node', node)
1271-
page_id_a2 = self.backup_node(
1269+
gdb = self.backup_node(
1270+
backup_dir, 'node', node, backup_type='page', gdb=True)
1271+
1272+
page_id_a3 = self.backup_node(
12721273
backup_dir, 'node', node, backup_type='page')
12731274

12741275
# Change FULLb backup status to ERROR
12751276
self.change_backup_status(backup_dir, 'node', backup_id_b, 'ERROR')
12761277

1278+
# Clean after yourself
1279+
self.del_test_dir(module_name, fname)
1280+
1281+
# @unittest.skip("skip")
1282+
def test_window_error_backups_1(self):
1283+
"""
1284+
DELTA
1285+
PAGE ERROR
1286+
FULL
1287+
-------window
1288+
"""
1289+
fname = self.id().split('.')[3]
1290+
node = self.make_simple_node(
1291+
base_dir=os.path.join(module_name, fname, 'node'),
1292+
initdb_params=['--data-checksums'])
1293+
1294+
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
1295+
self.init_pb(backup_dir)
1296+
self.add_instance(backup_dir, 'node', node)
1297+
self.set_archiving(backup_dir, 'node', node)
1298+
node.slow_start()
1299+
1300+
# Take FULL BACKUP
1301+
full_id = self.backup_node(backup_dir, 'node', node)
1302+
1303+
# Take PAGE BACKUP
1304+
gdb = self.backup_node(
1305+
backup_dir, 'node', node, backup_type='page', gdb=True)
1306+
1307+
gdb.set_breakpoint('pg_stop_backup')
1308+
gdb.run_until_break()
1309+
gdb.remove_all_breakpoints()
1310+
gdb._execute('signal SIGINT')
1311+
gdb.continue_execution_until_error()
1312+
1313+
page_id = self.show_pb(backup_dir, 'node')[1]['id']
1314+
1315+
# Take DELTA backup
1316+
delta_id = self.backup_node(
1317+
backup_dir, 'node', node, backup_type='delta',
1318+
options=['--retention-window=2', '--delete-expired'])
1319+
1320+
# Take FULL BACKUP
1321+
full2_id = self.backup_node(backup_dir, 'node', node)
1322+
1323+
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 4)
1324+
1325+
# Clean after yourself
1326+
self.del_test_dir(module_name, fname)
1327+
1328+
# @unittest.skip("skip")
1329+
def test_window_error_backups_2(self):
1330+
"""
1331+
DELTA
1332+
PAGE ERROR
1333+
FULL
1334+
-------window
1335+
"""
1336+
fname = self.id().split('.')[3]
1337+
node = self.make_simple_node(
1338+
base_dir=os.path.join(module_name, fname, 'node'),
1339+
initdb_params=['--data-checksums'])
1340+
1341+
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
1342+
self.init_pb(backup_dir)
1343+
self.add_instance(backup_dir, 'node', node)
1344+
self.set_archiving(backup_dir, 'node', node)
1345+
node.slow_start()
1346+
1347+
# Take FULL BACKUP
1348+
full_id = self.backup_node(backup_dir, 'node', node)
1349+
1350+
# Take PAGE BACKUP
1351+
gdb = self.backup_node(
1352+
backup_dir, 'node', node, backup_type='page', gdb=True)
1353+
1354+
gdb.set_breakpoint('pg_stop_backup')
1355+
gdb.run_until_break()
1356+
gdb._execute('signal SIGTERM')
1357+
gdb.continue_execution_until_error()
1358+
1359+
page_id = self.show_pb(backup_dir, 'node')[1]['id']
1360+
1361+
# Take DELTA backup
1362+
delta_id = self.backup_node(
1363+
backup_dir, 'node', node, backup_type='delta',
1364+
options=['--retention-window=2', '--delete-expired'])
1365+
1366+
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 3)
1367+
1368+
# Clean after yourself
1369+
# self.del_test_dir(module_name, fname)
1370+
1371+
def test_retention_redundancy_overlapping_chains(self):
1372+
""""""
1373+
fname = self.id().split('.')[3]
1374+
node = self.make_simple_node(
1375+
base_dir=os.path.join(module_name, fname, 'node'),
1376+
initdb_params=['--data-checksums'])
1377+
1378+
if self.get_version(node) < 90600:
1379+
self.del_test_dir(module_name, fname)
1380+
return unittest.skip('Skipped because ptrack support is disabled')
1381+
1382+
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
1383+
self.init_pb(backup_dir)
1384+
self.add_instance(backup_dir, 'node', node)
1385+
self.set_archiving(backup_dir, 'node', node)
1386+
node.slow_start()
1387+
1388+
self.set_config(
1389+
backup_dir, 'node', options=['--retention-redundancy=1'])
1390+
1391+
# Make backups to be purged
1392+
self.backup_node(backup_dir, 'node', node)
1393+
self.backup_node(backup_dir, 'node', node, backup_type="page")
1394+
1395+
# Make backups to be keeped
1396+
gdb = self.backup_node(backup_dir, 'node', node, gdb=True)
1397+
gdb.set_breakpoint('backup_files')
1398+
gdb.run_until_break()
1399+
1400+
sleep(1)
1401+
1402+
self.backup_node(backup_dir, 'node', node, backup_type="page")
1403+
1404+
gdb.remove_all_breakpoints()
1405+
gdb.continue_execution_until_exit()
1406+
1407+
self.backup_node(backup_dir, 'node', node, backup_type="page")
1408+
1409+
# Purge backups
1410+
log = self.delete_expired(
1411+
backup_dir, 'node', options=['--expired', '--wal'])
1412+
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 2)
1413+
1414+
# Clean after yourself
1415+
self.del_test_dir(module_name, fname)
1416+
12771417
def test_retention_redundancy_overlapping_chains(self):
12781418
""""""
12791419
fname = self.id().split('.')[3]
@@ -1319,3 +1459,49 @@ def test_retention_redundancy_overlapping_chains(self):
13191459

13201460
# Clean after yourself
13211461
self.del_test_dir(module_name, fname)
1462+
1463+
def test_agressive_retention_window_purge(self):
1464+
"""
1465+
https://github.com/postgrespro/pg_probackup/issues/106
1466+
"""
1467+
fname = self.id().split('.')[3]
1468+
node = self.make_simple_node(
1469+
base_dir=os.path.join(module_name, fname, 'node'),
1470+
initdb_params=['--data-checksums'])
1471+
1472+
backup_dir = os.path.join(self.tmp_path, module_name, fname, 'backup')
1473+
self.init_pb(backup_dir)
1474+
self.add_instance(backup_dir, 'node', node)
1475+
node.slow_start()
1476+
1477+
# Make ERROR incremental backup
1478+
try:
1479+
self.backup_node(backup_dir, 'node', node, backup_type='page')
1480+
# we should die here because exception is what we expect to happen
1481+
self.assertEqual(
1482+
1, 0,
1483+
"Expecting Error because page backup should not be possible "
1484+
"without valid full backup.\n Output: {0} \n CMD: {1}".format(
1485+
repr(self.output), self.cmd))
1486+
except ProbackupException as e:
1487+
self.assertIn(
1488+
"ERROR: Valid backup on current timeline 1 is not found. "
1489+
"Create new FULL backup before an incremental one.",
1490+
e.message,
1491+
"\n Unexpected Error Message: {0}\n CMD: {1}".format(
1492+
repr(e.message), self.cmd))
1493+
1494+
page_id = self.show_pb(backup_dir, 'node')[0]['id']
1495+
1496+
sleep(1)
1497+
1498+
# Make FULL backup
1499+
self.backup_node(
1500+
backup_dir, 'node', node,
1501+
options=['--delete-expired', '--retention-window=1', '--stream'])
1502+
1503+
# Check number of backups
1504+
self.assertEqual(len(self.show_pb(backup_dir, 'node')), 2)
1505+
1506+
# Clean after yourself
1507+
self.del_test_dir(module_name, fname)

0 commit comments

Comments
 (0)