Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 178dfd5

Browse files
CamStanadammoody
CamStan
authored and
adammoody
committed
Add stage/transfer API tests to CI framework
Creates lower-level tests (t/ci/800-stage-tests.sh) to do a proof of concept of the transfer API. Also creates a higher-level file (t/ci/RUN_CI_STAGE_TESTS.sh) that runs the lower-level tests over a sweep of file sizes and server configurations. Added a parallel transfer test to t/ci/800-stage-tests.sh as well. It is currently being skipped until the parallel transfer logic is fixed. Fixes a bug in serial transfer logic causing the destination file to be created as read-only, resulting in a failed transfer. Update transfer docs to account for the various methods.
1 parent 9058fe7 commit 178dfd5

File tree

6 files changed

+610
-78
lines changed

6 files changed

+610
-78
lines changed

‎client/src/posix_client.c‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -872,12 +872,24 @@ int unifyfs_transfer_file(const char* src,
872872
return -EINVAL;
873873
}
874874

875-
/* for both serial and parallel transfers, use rank 0 client to
876-
* create the destination file using the source file's mode */
875+
/* TODO: Fix parallel transfer logic
876+
* for both serial and parallel transfers, use rank 0 client to
877+
* create the destination file */
877878
if (0 == client_rank) {
878879
errno = 0;
879880
int create_flags = O_CREAT | O_WRONLY | O_TRUNC;
880-
int fd = UNIFYFS_WRAP(open)(dst_path, create_flags, sb_src.st_mode);
881+
int dst_mode;
882+
883+
if (unify_src) {
884+
/* Destination file needs to be writable; file in UnifyFS may have
885+
* been laminated */
886+
dst_mode = 0640;
887+
} else {
888+
/* Use the source file's mode */
889+
dst_mode = sb_src.st_mode;
890+
}
891+
892+
int fd = UNIFYFS_WRAP(open)(dst_path, create_flags, dst_mode);
881893
err = errno;
882894
if (fd < 0) {
883895
LOGERR("failed to create destination file %s", dst_path);

‎docs/run.rst‎

Lines changed: 196 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ Overall, the steps to run an application with UnifyFS include:
1717

1818
5. Terminate the UnifyFS servers using ``unifyfs``
1919

20-
--------------------
21-
Start UnifyFS
22-
--------------------
20+
-------------
21+
Start UnifyFS
22+
-------------
2323

2424
First, one must start the UnifyFS server process (``unifyfsd``) on the nodes in
2525
the job allocation. UnifyFS provides the ``unifyfs`` command line utility to
@@ -57,7 +57,6 @@ adjust the consistency model, and control stage-in and stage-out of files.
5757
The full usage for ``unifyfs`` is as follows:
5858

5959
.. code-block:: Bash
60-
:linenos:
6160
6261
[prompt]$ unifyfs --help
6362
@@ -72,18 +71,21 @@ The full usage for ``unifyfs`` is as follows:
7271
-h, --help print usage
7372
7473
Command options for "start":
75-
-C, --consistency=<model> [OPTIONAL] consistency model (NONE | LAMINATED | POSIX)
76-
-e, --exe=<path> [OPTIONAL] <path> where unifyfsd is installed
77-
-m, --mount=<path> [OPTIONAL] mount UnifyFS at <path>
78-
-s, --script=<path> [OPTIONAL] <path> to custom launch script
79-
-t, --timeout=<sec> [OPTIONAL] wait <sec> until all servers become ready
80-
-S, --share-dir=<path> [REQUIRED] shared file system <path> for use by servers
81-
-c, --cleanup [OPTIONAL] clean up the UnifyFS storage upon server exit
82-
-i, --stage-in=<path> [OPTIONAL] stage in manifest file(s) at <path>
74+
-C, --consistency=<model> [OPTIONAL] consistency model (NONE | LAMINATED | POSIX)
75+
-e, --exe=<path> [OPTIONAL] <path> where unifyfsd is installed
76+
-m, --mount=<path> [OPTIONAL] mount UnifyFS at <path>
77+
-s, --script=<path> [OPTIONAL] <path> to custom launch script
78+
-t, --timeout=<sec> [OPTIONAL] wait <sec> until all servers become ready
79+
-S, --share-dir=<path> [REQUIRED] shared file system <path> for use by servers
80+
-c, --cleanup [OPTIONAL] clean up the UnifyFS storage upon server exit
81+
-i, --stage-in=<manifest> [OPTIONAL] stage in file(s) listed in <manifest> file
82+
-T, --stage-timeout=<sec> [OPTIONAL] timeout for stage-in operation
8383
8484
Command options for "terminate":
85-
-s, --script=<path> [OPTIONAL] <path> to custom termination script
86-
-o, --stage-out=<path> [OPTIONAL] stage out manifest file(s) at <path>
85+
-o, --stage-out=<manifest> [OPTIONAL] stage out file(s) listed in <manifest> on termination
86+
-T, --stage-timeout=<sec> [OPTIONAL] timeout for stage-out operation
87+
-s, --script=<path> [OPTIONAL] <path> to custom termination script
88+
-S, --share-dir=<path> [REQUIRED for --stage-out] shared file system <path> for use by servers
8789
8890
8991
After UnifyFS servers have been successfully started, you may run your
@@ -92,18 +94,18 @@ Only applications that explicitly call ``unifyfs_mount()`` and access files
9294
under the specified mountpoint prefix will utilize UnifyFS for their I/O. All
9395
other applications will operate unchanged.
9496
95-
--------------------
96-
Stop UnifyFS
97-
--------------------
97+
------------
98+
Stop UnifyFS
99+
------------
98100
99-
After all UnifyFS-enabled applications have completed running, you should
100-
use ``unifyfs terminate`` to terminate the servers. Typically, one would pass
101-
the ``--cleanup`` option to ``unifyfs start`` to have the servers remove
102-
temporary data locally stored on each node after termination.
101+
After all UnifyFS-enabled applications have completed running, use
102+
``unifyfs terminate`` to terminate the servers. Pass the ``--cleanup`` option to
103+
``unifyfs start`` to have the servers remove temporary data locally stored on
104+
each node after termination.
103105
104-
------------------------------------
105-
Resource Manager Job Integration
106-
------------------------------------
106+
--------------------------------
107+
Resource Manager Job Integration
108+
--------------------------------
107109
108110
UnifyFS includes optional support for integrating directly with compatible
109111
resource managers to automatically start and stop servers at the beginning
@@ -118,28 +120,176 @@ within the source repository at ``util/scripts/lsfcsm``.
118120
119121
Support for the SLURM resource manager is under development.
120122
121-
-----------------------------------------------
122-
Stage-in and Stage-out Manifest File Format
123-
-----------------------------------------------
124-
125-
The manifest file contains one or more file copy requests.
126-
Each line in the manifest corresponds to one file copy request,
127-
and it contains both the source and destination file paths. Currently,
128-
directory copies are not supported.
129-
130-
Each line is formatted as: ``<source filename> <whitespace> <destination filename>``.
131-
If either of the filenames
132-
contain whitespace or special characters, then both filenames should
133-
be surrounded by double-quote characters (") (ASCII character 34 decimal).
134-
The double-quote character and the linefeed end-of-line character are forbidden
135-
in any filenames used in a unifyfs manifest file, but any other
136-
characters are allowed, including control characters.
137-
If a filename contains any characters that might be misinterpreted, then
138-
enclosing the filename in double-quotes is always
139-
a safe thing to do.
123+
-----------------------------------------
124+
Transferring Data In and Out of UnifyFS
125+
-----------------------------------------
126+
127+
Data can be transferred in/out of UnifyFS during server startup and termination,
128+
or at any point during a job using two stand-alone applications.
129+
130+
Transfer at Server Start/Terminate
131+
**********************************
132+
133+
The transfer subsystem within UnifyFS can be invoked by providing the
134+
``-i|--stage-in`` option to ``unifyfs start`` to transfer files into UnifyFS:
135+
136+
.. code-block:: Bash
137+
138+
$ unifyfs start --stage-in=/path/to/input/manifest/file --share-dir=/path/to/shared/file/system
139+
140+
and/or by providing the ``-o|--stage-out``, and consequently required
141+
``-S|--share-dir``, option to ``unifyfs terminate`` to transfer files out of
142+
UnifyFS:
143+
144+
.. code-block:: Bash
145+
146+
$ unifyfs terminate --stage-out=/path/to/output/manifest/file --share-dir=/path/to/shared/file/system
147+
148+
A manifest file needs to be provided to the ``start``/``terminate`` commands in
149+
order to specify the desired transfers.
150+
151+
.. _manifest_file_label:
152+
153+
Manifest File
154+
^^^^^^^^^^^^^
155+
156+
UnifyFS's stage functionality requires a manifest file in order to move data.
157+
158+
The manifest file contains one or more file copy requests. Each line in the
159+
manifest corresponds to one transfer request, and it contains both the source
160+
and destination file paths. Directory copies are currently not supported.
161+
162+
Each line is formatted as:
163+
``<source filename> <whitespace> <destination filename>``.
164+
165+
If either of the filenames contain whitespace or special characters, then both
166+
filenames should be surrounded by double-quote characters (") (ASCII character
167+
34 decimal).
168+
The double-quote and linefeed end-of-line characters are not supported in any
169+
filenames used in a unifyfs manifest file. Any other characters are allowed,
170+
including control characters. If a filename contains any characters that might
171+
be misinterpreted, then enclosing the filename in double-quotes is always a safe
172+
thing to do.
140173
141174
Here is an example of a valid stage-in manifest file:
142175
143-
``/scratch/users/me/input_data/input_1.dat /unifyfs/input/input_1.dat``
144-
``/home/users/me/configuration/run_12345.conf /unifyfs/config/run_12345.conf``
145-
``"/home/users/me/file with space.dat" "/unifyfs/file with space.dat"``
176+
.. code-block:: Bash
177+
178+
$ [prompt] cat example_stage_in.manifest
179+
180+
/scratch/users/me/input_data/input_1.dat /unifyfs/input/input_1.dat
181+
/home/users/me/configuration/run_12345.conf /unifyfs/config/run_12345.conf
182+
"/home/users/me/file with space.dat" "/unifyfs/file with space.dat"
183+
184+
Transfer During Job
185+
*******************
186+
187+
Data can also be transferred in/out of UnifyFS using one of two stand-alone
188+
applications.
189+
190+
The stand-alone applications can be invoked at any time while the UnifyFS
191+
servers are up and responding to requests. This allows for bringing in new input
192+
and/or transferring results out to be verified before the job terminates.
193+
194+
UnifyFS Stage Executable
195+
^^^^^^^^^^^^^^^^^^^^^^^^
196+
197+
``$UNIFYFS_INSTALL/libexec/unifyfs-stage``
198+
199+
The ``start``/``terminate`` transfer API stage functionality can also be used
200+
via the stand-alone application ``unifyfs-stage``.
201+
202+
This application can be run at any time within a job to transfer new data into
203+
or results out of UnifyFS.
204+
A manifest file (see :ref:`above <manifest_file_label>`_) needs to be provided
205+
as an argument to use this approach.
206+
207+
.. code-block:: Bash
208+
209+
[prompt]$ ./unifyfs-stage --help
210+
211+
Usage: unifyfs-stage [OPTION]... <manifest file>
212+
213+
Transfer files between unifyfs volume and external file system.
214+
The <manifest file> should contain list of files to be transferred,
215+
and each line should be formatted as
216+
217+
/source/file/path /destination/file/path
218+
219+
OR in the case of filenames with spaces or special characters:
220+
221+
"/source/file/path" "/destination/file/path"
222+
223+
One file per line; Specifying directories is not supported.
224+
225+
Available options:
226+
-c, --checksum verify md5 checksum for each transfer
227+
-h, --help print this usage
228+
-m, --mountpoint=<mnt> use <mnt> as unifyfs mountpoint
229+
(default: /unifyfs)
230+
-p, --parallel transfer each file in parallel
231+
(experimental)
232+
-s, --share-dir=<path> directory path for creating status file
233+
-v, --verbose print noisy outputs
234+
235+
Without the '-p, --parallel' option, a file is transferred by a single
236+
process. If the '-p, --parallel' option is specified, each file will be
237+
divided by multiple processes and transferred in parallel.
238+
239+
Examples:
240+
241+
.. code-block:: Bash
242+
:caption: Serial Transfer
243+
244+
$ srun -N 1 -n 1 $UNIFYFS_INSTALL/libexec/unifyfs-stage $MY_MANIFEST_FILE
245+
246+
.. code-block:: Bash
247+
:caption: Parallel Transfer
248+
249+
$ srun -N 4 -n 8 $UNIFYFS_INSTALL/libexec/unifyfs-stage --parallel $MY_MANIFEST_FILE
250+
251+
Transfer Executable
252+
^^^^^^^^^^^^^^^^^^^
253+
254+
``$UNIFYFS_INSTALL/libexec/transfer-static``
255+
256+
.. note::
257+
258+
The ``transfer-gotcha`` executable is currently unusable due to an issue
259+
that is being tracked.
260+
261+
The transfer API can also be used during the job by invoking the stand-alone
262+
``transfer`` application. It works similarly to the Unix ``cp`` command, with
263+
source and destination, except being aware that it is copying files between an
264+
external file system and internal UnifyFS.
265+
266+
.. code-block:: Bash
267+
268+
[prompt]$ transfer-static --help
269+
270+
Usage: transfer-static [options...] <source path> <destination path>
271+
272+
Available options:
273+
-d, --debug pause before running test
274+
(handy for attaching in debugger)
275+
-h, --help help message
276+
-m, --mount=<mountpoint> use <mountpoint> for unifyfs
277+
(default: /unifyfs)
278+
-p, --parallel parallel transfer
279+
-r, --rank=<rank> use <rank> for transfer (default: 0)
280+
281+
Examples of using ``transfer-static``:
282+
283+
.. code-block:: Bash
284+
:caption: Serial Transfer
285+
286+
$ srun -N 1 -n 1 $UNIFYFS_INSTALL/libexec/transfer-static /path/on/parallelfs/file.dat /unifyfs/file.dat
287+
288+
$ srun -N 1 -n 1 $UNIFYFS_INSTALL/libexec/transfer-static /unifyfs/output.dat /scratch/my_output/output.dat
289+
290+
.. code-block:: Bash
291+
:caption: Parallel Transfer
292+
293+
$ srun -N 4 -n 8 /path/to/libexec/transfer-static -parallel /path/on/parallelfs/file.dat /unifyfs/file.dat
294+
295+
$ srun -N 4 -n 8 /path/to/libexec/transfer-static -parallel /unifyfs/output.dat /scratch/my_output/output.dat

0 commit comments

Comments
 (0)