Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 8c16ad3

Browse files
nathan-bossartnathan-bossart
committed
Allow using syncfs() in frontend utilities.
This commit allows specifying a --sync-method in several frontend utilities that must synchronize many files to disk (initdb, pg_basebackup, pg_checksums, pg_dump, pg_rewind, and pg_upgrade). On Linux, users can specify "syncfs" to synchronize the relevant file systems instead of calling fsync() for every single file. In many cases, using syncfs() is much faster. As with recovery_init_sync_method, this new option comes with some caveats. The descriptions of these caveats have been moved to a new appendix section in the documentation. Co-authored-by: Justin Pryzby Reviewed-by: Michael Paquier, Thomas Munro, Robert Haas, Justin Pryzby Discussion: https://postgr.es/m/20210930004340.GM831%40telsasoft.com
1 parent cccc6cd commit 8c16ad3

21 files changed

+271
-11
lines changed

‎doc/src/sgml/config.sgml

+3-9
Original file line numberDiff line numberDiff line change
@@ -10511,15 +10511,9 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
1051110511
On Linux, <literal>syncfs</literal> may be used instead, to ask the
1051210512
operating system to synchronize the whole file systems that contain the
1051310513
data directory, the WAL files and each tablespace (but not any other
10514-
file systems that may be reachable through symbolic links). This may
10515-
be a lot faster than the <literal>fsync</literal> setting, because it
10516-
doesn't need to open each file one by one. On the other hand, it may
10517-
be slower if a file system is shared by other applications that
10518-
modify a lot of files, since those files will also be written to disk.
10519-
Furthermore, on versions of Linux before 5.8, I/O errors encountered
10520-
while writing data to disk may not be reported to
10521-
<productname>PostgreSQL</productname>, and relevant error messages may
10522-
appear only in kernel logs.
10514+
file systems that may be reachable through symbolic links). See
10515+
<xref linkend="syncfs"/> for more information about using
10516+
<function>syncfs()</function>.
1052310517
</para>
1052410518
<para>
1052510519
This parameter can only be set in the

‎doc/src/sgml/filelist.sgml

+1
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@
183183
<!ENTITY acronyms SYSTEM "acronyms.sgml">
184184
<!ENTITY glossary SYSTEM "glossary.sgml">
185185
<!ENTITY color SYSTEM "color.sgml">
186+
<!ENTITY syncfs SYSTEM "syncfs.sgml">
186187

187188
<!ENTITY features-supported SYSTEM "features-supported.sgml">
188189
<!ENTITY features-unsupported SYSTEM "features-unsupported.sgml">

‎doc/src/sgml/postgres.sgml

+1
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,7 @@ break is not needed in a wider output rendering.
294294
&acronyms;
295295
&glossary;
296296
&color;
297+
&syncfs;
297298
&obsolete;
298299

299300
</part>

‎doc/src/sgml/ref/initdb.sgml

+22
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,28 @@ PostgreSQL documentation
365365
</listitem>
366366
</varlistentry>
367367

368+
<varlistentry id="app-initdb-option-sync-method">
369+
<term><option>--sync-method</option></term>
370+
<listitem>
371+
<para>
372+
When set to <literal>fsync</literal>, which is the default,
373+
<command>initdb</command> will recursively open and synchronize all
374+
files in the data directory. The search for files will follow symbolic
375+
links for the WAL directory and each configured tablespace.
376+
</para>
377+
<para>
378+
On Linux, <literal>syncfs</literal> may be used instead to ask the
379+
operating system to synchronize the whole file systems that contain the
380+
data directory, the WAL files, and each tablespace. See
381+
<xref linkend="syncfs"/> for more information about using
382+
<function>syncfs()</function>.
383+
</para>
384+
<para>
385+
This option has no effect when <option>--no-sync</option> is used.
386+
</para>
387+
</listitem>
388+
</varlistentry>
389+
368390
<varlistentry id="app-initdb-option-sync-only">
369391
<term><option>-S</option></term>
370392
<term><option>--sync-only</option></term>

‎doc/src/sgml/ref/pg_basebackup.sgml

+25
Original file line numberDiff line numberDiff line change
@@ -594,6 +594,31 @@ PostgreSQL documentation
594594
</listitem>
595595
</varlistentry>
596596

597+
<varlistentry>
598+
<term><option>--sync-method</option></term>
599+
<listitem>
600+
<para>
601+
When set to <literal>fsync</literal>, which is the default,
602+
<command>pg_basebackup</command> will recursively open and synchronize
603+
all files in the backup directory. When the plain format is used, the
604+
search for files will follow symbolic links for the WAL directory and
605+
each configured tablespace.
606+
</para>
607+
<para>
608+
On Linux, <literal>syncfs</literal> may be used instead to ask the
609+
operating system to synchronize the whole file system that contains the
610+
backup directory. When the plain format is used,
611+
<command>pg_basebackup</command> will also synchronize the file systems
612+
that contain the WAL files and each tablespace. See
613+
<xref linkend="syncfs"/> for more information about using
614+
<function>syncfs()</function>.
615+
</para>
616+
<para>
617+
This option has no effect when <option>--no-sync</option> is used.
618+
</para>
619+
</listitem>
620+
</varlistentry>
621+
597622
<varlistentry>
598623
<term><option>-v</option></term>
599624
<term><option>--verbose</option></term>

‎doc/src/sgml/ref/pg_checksums.sgml

+22
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,28 @@ PostgreSQL documentation
139139
</listitem>
140140
</varlistentry>
141141

142+
<varlistentry>
143+
<term><option>--sync-method</option></term>
144+
<listitem>
145+
<para>
146+
When set to <literal>fsync</literal>, which is the default,
147+
<command>pg_checksums</command> will recursively open and synchronize
148+
all files in the data directory. The search for files will follow
149+
symbolic links for the WAL directory and each configured tablespace.
150+
</para>
151+
<para>
152+
On Linux, <literal>syncfs</literal> may be used instead to ask the
153+
operating system to synchronize the whole file systems that contain the
154+
data directory, the WAL files, and each tablespace. See
155+
<xref linkend="syncfs"/> for more information about using
156+
<function>syncfs()</function>.
157+
</para>
158+
<para>
159+
This option has no effect when <option>--no-sync</option> is used.
160+
</para>
161+
</listitem>
162+
</varlistentry>
163+
142164
<varlistentry>
143165
<term><option>-v</option></term>
144166
<term><option>--verbose</option></term>

‎doc/src/sgml/ref/pg_dump.sgml

+21
Original file line numberDiff line numberDiff line change
@@ -1179,6 +1179,27 @@ PostgreSQL documentation
11791179
</listitem>
11801180
</varlistentry>
11811181

1182+
<varlistentry>
1183+
<term><option>--sync-method</option></term>
1184+
<listitem>
1185+
<para>
1186+
When set to <literal>fsync</literal>, which is the default,
1187+
<command>pg_dump --format=directory</command> will recursively open and
1188+
synchronize all files in the archive directory.
1189+
</para>
1190+
<para>
1191+
On Linux, <literal>syncfs</literal> may be used instead to ask the
1192+
operating system to synchronize the whole file system that contains the
1193+
archive directory. See <xref linkend="syncfs"/> for more information
1194+
about using <function>syncfs()</function>.
1195+
</para>
1196+
<para>
1197+
This option has no effect when <option>--no-sync</option> is used or
1198+
<option>--format</option> is not set to <literal>directory</literal>.
1199+
</para>
1200+
</listitem>
1201+
</varlistentry>
1202+
11821203
<varlistentry>
11831204
<term><option>--table-and-children=<replaceable class="parameter">pattern</replaceable></option></term>
11841205
<listitem>

‎doc/src/sgml/ref/pg_rewind.sgml

+22
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,28 @@ PostgreSQL documentation
284284
</listitem>
285285
</varlistentry>
286286

287+
<varlistentry>
288+
<term><option>--sync-method</option></term>
289+
<listitem>
290+
<para>
291+
When set to <literal>fsync</literal>, which is the default,
292+
<command>pg_rewind</command> will recursively open and synchronize all
293+
files in the data directory. The search for files will follow symbolic
294+
links for the WAL directory and each configured tablespace.
295+
</para>
296+
<para>
297+
On Linux, <literal>syncfs</literal> may be used instead to ask the
298+
operating system to synchronize the whole file systems that contain the
299+
data directory, the WAL files, and each tablespace. See
300+
<xref linkend="syncfs"/> for more information about using
301+
<function>syncfs()</function>.
302+
</para>
303+
<para>
304+
This option has no effect when <option>--no-sync</option> is used.
305+
</para>
306+
</listitem>
307+
</varlistentry>
308+
287309
<varlistentry>
288310
<term><option>-V</option></term>
289311
<term><option>--version</option></term>

‎doc/src/sgml/ref/pgupgrade.sgml

+23
Original file line numberDiff line numberDiff line change
@@ -190,6 +190,29 @@ PostgreSQL documentation
190190
variable <envar>PGSOCKETDIR</envar></para></listitem>
191191
</varlistentry>
192192

193+
<varlistentry>
194+
<term><option>--sync-method</option></term>
195+
<listitem>
196+
<para>
197+
When set to <literal>fsync</literal>, which is the default,
198+
<command>pg_upgrade</command> will recursively open and synchronize all
199+
files in the upgraded cluster's data directory. The search for files
200+
will follow symbolic links for the WAL directory and each configured
201+
tablespace.
202+
</para>
203+
<para>
204+
On Linux, <literal>syncfs</literal> may be used instead to ask the
205+
operating system to synchronize the whole file systems that contain the
206+
upgraded cluster's data directory, its WAL files, and each tablespace.
207+
See <xref linkend="syncfs"/> for more information about using
208+
<function>syncfs()</function>.
209+
</para>
210+
<para>
211+
This option has no effect when <option>--no-sync</option> is used.
212+
</para>
213+
</listitem>
214+
</varlistentry>
215+
193216
<varlistentry>
194217
<term><option>-U</option> <replaceable>username</replaceable></term>
195218
<term><option>--username=</option><replaceable>username</replaceable></term>

‎doc/src/sgml/syncfs.sgml

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
<!-- doc/src/sgml/syncfs.sgml -->
2+
3+
<appendix id="syncfs">
4+
<title><function>syncfs()</function> Caveats</title>
5+
6+
<indexterm zone="syncfs">
7+
<primary>syncfs</primary>
8+
</indexterm>
9+
10+
<para>
11+
On Linux <function>syncfs()</function> may be specified for some
12+
configuration parameters (e.g.,
13+
<xref linkend="guc-recovery-init-sync-method"/>), server applications (e.g.,
14+
<application>pg_upgrade</application>), and client applications (e.g.,
15+
<application>pg_basebackup</application>) that involve synchronizing many
16+
files to disk. <function>syncfs()</function> is advantageous in many cases,
17+
but there are some trade-offs to keep in mind.
18+
</para>
19+
20+
<para>
21+
Since <function>syncfs()</function> instructs the operating system to
22+
synchronize a whole file system, it typically requires many fewer system
23+
calls than using <function>fsync()</function> to synchronize each file one by
24+
one. Therefore, using <function>syncfs()</function> may be a lot faster than
25+
using <function>fsync()</function>. However, it may be slower if a file
26+
system is shared by other applications that modify a lot of files, since
27+
those files will also be written to disk.
28+
</para>
29+
30+
<para>
31+
Furthermore, on versions of Linux before 5.8, I/O errors encountered while
32+
writing data to disk may not be reported to the calling program, and relevant
33+
error messages may appear only in kernel logs.
34+
</para>
35+
36+
</appendix>

‎src/bin/initdb/initdb.c

+6
Original file line numberDiff line numberDiff line change
@@ -2467,6 +2467,7 @@ usage(const char *progname)
24672467
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
24682468
printf(_(" --no-instructions do not print instructions for next steps\n"));
24692469
printf(_(" -s, --show show internal settings\n"));
2470+
printf(_(" --sync-method=METHOD set method for syncing files to disk\n"));
24702471
printf(_(" -S, --sync-only only sync database files to disk, then exit\n"));
24712472
printf(_("\nOther options:\n"));
24722473
printf(_(" -V, --version output version information, then exit\n"));
@@ -3107,6 +3108,7 @@ main(int argc, char *argv[])
31073108
{"locale-provider", required_argument, NULL, 15},
31083109
{"icu-locale", required_argument, NULL, 16},
31093110
{"icu-rules", required_argument, NULL, 17},
3111+
{"sync-method", required_argument, NULL, 18},
31103112
{NULL, 0, NULL, 0}
31113113
};
31123114

@@ -3287,6 +3289,10 @@ main(int argc, char *argv[])
32873289
case 17:
32883290
icu_rules = pg_strdup(optarg);
32893291
break;
3292+
case 18:
3293+
if (!parse_sync_method(optarg, &sync_method))
3294+
exit(1);
3295+
break;
32903296
default:
32913297
/* getopt_long already emitted a complaint */
32923298
pg_log_error_hint("Try \"%s --help\" for more information.", progname);

‎src/bin/initdb/t/001_initdb.pl

+12
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
my $tempdir = PostgreSQL::Test::Utils::tempdir;
1717
my $xlogdir = "$tempdir/pgxlog";
1818
my $datadir = "$tempdir/data";
19+
my $supports_syncfs = check_pg_config("#define HAVE_SYNCFS 1");
1920

2021
program_help_ok('initdb');
2122
program_version_ok('initdb');
@@ -82,6 +83,17 @@
8283
command_ok([ 'initdb', '-S', $datadir ], 'sync only');
8384
command_fails([ 'initdb', $datadir ], 'existing data directory');
8485

86+
if ($supports_syncfs)
87+
{
88+
command_ok([ 'initdb', '-S', $datadir, '--sync-method', 'syncfs' ],
89+
'sync method syncfs');
90+
}
91+
else
92+
{
93+
command_fails([ 'initdb', '-S', $datadir, '--sync-method', 'syncfs' ],
94+
'sync method syncfs');
95+
}
96+
8597
# Check group access on PGDATA
8698
SKIP:
8799
{

‎src/bin/pg_basebackup/pg_basebackup.c

+7
Original file line numberDiff line numberDiff line change
@@ -425,6 +425,8 @@ usage(void)
425425
printf(_(" --no-slot prevent creation of temporary replication slot\n"));
426426
printf(_(" --no-verify-checksums\n"
427427
" do not verify checksums\n"));
428+
printf(_(" --sync-method=METHOD\n"
429+
" set method for syncing files to disk\n"));
428430
printf(_(" -?, --help show this help, then exit\n"));
429431
printf(_("\nConnection options:\n"));
430432
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -2282,6 +2284,7 @@ main(int argc, char **argv)
22822284
{"no-manifest", no_argument, NULL, 5},
22832285
{"manifest-force-encode", no_argument, NULL, 6},
22842286
{"manifest-checksums", required_argument, NULL, 7},
2287+
{"sync-method", required_argument, NULL, 8},
22852288
{NULL, 0, NULL, 0}
22862289
};
22872290
int c;
@@ -2453,6 +2456,10 @@ main(int argc, char **argv)
24532456
case 7:
24542457
manifest_checksums = pg_strdup(optarg);
24552458
break;
2459+
case 8:
2460+
if (!parse_sync_method(optarg, &sync_method))
2461+
exit(1);
2462+
break;
24562463
default:
24572464
/* getopt_long already emitted a complaint */
24582465
pg_log_error_hint("Try \"%s --help\" for more information.", progname);

‎src/bin/pg_checksums/pg_checksums.c

+6
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,7 @@ usage(void)
7878
printf(_(" -f, --filenode=FILENODE check only relation with specified filenode\n"));
7979
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
8080
printf(_(" -P, --progress show progress information\n"));
81+
printf(_(" --sync-method=METHOD set method for syncing files to disk\n"));
8182
printf(_(" -v, --verbose output verbose messages\n"));
8283
printf(_(" -V, --version output version information, then exit\n"));
8384
printf(_(" -?, --help show this help, then exit\n"));
@@ -436,6 +437,7 @@ main(int argc, char *argv[])
436437
{"no-sync", no_argument, NULL, 'N'},
437438
{"progress", no_argument, NULL, 'P'},
438439
{"verbose", no_argument, NULL, 'v'},
440+
{"sync-method", required_argument, NULL, 1},
439441
{NULL, 0, NULL, 0}
440442
};
441443

@@ -494,6 +496,10 @@ main(int argc, char *argv[])
494496
case 'v':
495497
verbose = true;
496498
break;
499+
case 1:
500+
if (!parse_sync_method(optarg, &sync_method))
501+
exit(1);
502+
break;
497503
default:
498504
/* getopt_long already emitted a complaint */
499505
pg_log_error_hint("Try \"%s --help\" for more information.", progname);

‎src/bin/pg_dump/pg_dump.c

+7
Original file line numberDiff line numberDiff line change
@@ -432,6 +432,7 @@ main(int argc, char **argv)
432432
{"table-and-children", required_argument, NULL, 12},
433433
{"exclude-table-and-children", required_argument, NULL, 13},
434434
{"exclude-table-data-and-children", required_argument, NULL, 14},
435+
{"sync-method", required_argument, NULL, 15},
435436

436437
{NULL, 0, NULL, 0}
437438
};
@@ -658,6 +659,11 @@ main(int argc, char **argv)
658659
optarg);
659660
break;
660661

662+
case 15:
663+
if (!parse_sync_method(optarg, &sync_method))
664+
exit_nicely(1);
665+
break;
666+
661667
default:
662668
/* getopt_long already emitted a complaint */
663669
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -1069,6 +1075,7 @@ help(const char *progname)
10691075
" compress as specified\n"));
10701076
printf(_(" --lock-wait-timeout=TIMEOUT fail after waiting TIMEOUT for a table lock\n"));
10711077
printf(_(" --no-sync do not wait for changes to be written safely to disk\n"));
1078+
printf(_(" --sync-method=METHOD set method for syncing files to disk\n"));
10721079
printf(_(" -?, --help show this help, then exit\n"));
10731080

10741081
printf(_("\nOptions controlling the output content:\n"));

0 commit comments

Comments
 (0)