Forensic Science International: Digital Investigation 32 (2020) 300915

Contents lists available at ScienceDirect

Forensic Science International: Digital Investigation

journal homepage: www.elsevier.com/locate/fsidi

DFRWS 2020 EU e Proceedings of the Seventh Annual DFRWS Europe

Forensic Analysis of the Resilient File System (ReFS) Version 3.4

Paul Prade a, Tobias Grob a, *, Andreas Dewald b, a, **
a
Friedrich-Alexander University Erlangen-Nürnberg, Germany
b
ERNW Research GmbH, Heidelberg, Germany

a r t i c l e i n f o a b s t r a c t

Article history: ReFS is a modern file system that is developed by Microsoft and its internal structures and behavior is not
officially documented. Even so there exist some analysis efforts in deciphering its data structures, some
of these findings have yet become deprecated and cannot be applied to current ReFS versions anymore.
Keywords: In this work, general concepts and internal structures found in ReFS are examined and documented.
Digital forensics Based on the structures and the processes by which they are modified, approaches to recover (deleted)
Data recovery
files from ReFS formatted file systems are shown. We also evaluated our implementation and the allo-
File systems
cation strategy of ReFS with respect to accuracy, runtime and the ability to recover older file states.
ReFS
© 2020 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction analyzed ReFS before, but they looked into versions 1.1 and 1.2.
Some core concepts still exist in the latest version 3.4 but also major
Storage media analysis is a common task in the field of digital changes were applied between these versions, which draws older
forensic, when PCs or mobile devices get analyzed. Investigators work incompatible to newer versions of ReFS.
have to rely on the proper functioning of their tools to provide them Sinofsky (2012) explains key goals as well as features like
with correct interpretation of traces. File systems have to be checksums for meta- and content-data as well as the Copy-On-
interpreted and presented when analyzing storage media, doing Write (COW) mechanism that are provided by ReFS. He also men-
this manually is unfeasible. tions a generic key-value interface which is used by ReFS to manage
From this situation emerges the need for digital forensic tools to on-disk structures.
ideally support all of the file systems that are currently in use and In a working report, Green (2013) analyzed the internal struc-
may be encountered in a forensic analysis. Limitations of classical tures of ReFS and created an overview of identified structures. This
file systems such as low performance, limited capacity or unsuit- report skipped how the internal key-value structure of ReFS works.
ability for SSD drives led to the development of new filesystems like He also looked into the recovery of deleted items, but only de-
APFS or ReFS. These new filesystems have to be supported in open scribes the recycle bin mechanism of the windows file explorer
source forensic tools and documentation. Transparency in forensic which is not specific to ReFS.
processes and tools is important when digital evidence is used in Another unofficial draft of structures found under ReFS is pro-
severe cases. That is even more important when filesystems are vided by Metz (2013). This work strongly focuses on the low-level
proprietary as the above-mentioned ones. presentation of how data structures in ReFS are composed and was
With this work, we want to provide the forensic community the first to vaguely describe the key-value store. The author
with tools and information to properly analyze ReFS partitions. implemented a library which is based on his findings.
Georges (2018) aimed to develop a tool which outputs results
comparable to EnCase. He also described how the allocation status
1.1. Related work
of clusters is managed. With the developed tool, file extraction is
possible for ReFS v1.2.
There exists related work that we show in the following which
Nordvik et al. (2019) also examined data structures of ReFS. They
mainly focused on v1.2 and v3.2 of ReFS. At the end they tested their
* Corresponding author.
findings on v3.4, too. For v3.2 they came to the conclusion that the
** Corresponding author. Friedrich-Alexander University Erlangen-Nürnberg, versions did not differ much from v1.2 and that “the structures are
Germany. almost identical”. In our work we come to the conclusion that
E-mail addresses: [email protected] (T. Grob), [email protected] (A. Dewald).

https://doi.org/10.1016/j.fsidi.2020.300915
2666-2817/© 2020 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/
).
S2 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915

between v3.4 and v1.2 many data structures were added, depre- current clock values, which indicates the latest checkpoint struc-
cated or changed. We found also new functionality like virtual ture. Additionally, the checkpoint also contains the current log
addresses which have to be taken into account when extracting sequence number (LSN) which identifies the last entry that was
data from the disk. In contrast to their work, we discovered more written into the redo log. Since ReFS stores changes to the file
details and also investigate how the Copy-On-Write mechanism of system in the memory and writes them at a later point as a batch,
ReFS is implemented and propose strategies to recover old versions the system may crash in this time window. A system crash reverts
of files. the system to its last valid state and would discard all changes
executed in the meantime. However, every transaction that was
1.2. Our contribution successfully performed and is part of the batch is also written to a
sequential log file that is discussed in our report. To know from
In this work, we analyzed the proprietary Microsoft Windows where to start with the redo operations, it is essential to save an
Resilient File System (ReFS) version 3.4 and contribute the identifier for the last transaction that was part of the current
following: checkpoint.
The most important structures found in the checkpoint are the

We analyzed the internal structures and mechanics of ReFS v3.4 table references. For the sake of thoroughness, Table 1 presents an
and documented them in a technical report (Prade et al. (2019)). overview of all tables referenced by the checkpoint along with their

We extended The Sleuth Kit from Carrier to support ReFS. table identifiers.

We propose strategies for recovering deleted files.

We implemented a page carver which allows the reconstruction
of deleted files and older file states. 3.2. Container table

We evaluated the correctness of our implementation with
different ReFS partitions. Internally a ReFS volume is separated into multiple equally sized

We analyzed the allocation strategy of ReFS v3.4 which affects bands or containers. The size of a single band is relatively large at
the recoverability of older file states and compared the findings around 64 MiB or 256 MiB. The Container Table provides detailed
with our carver. information on that managed memory regions. It tracks how many
clusters within a band are used and additionally collects statistics
2. Background about the usage of data and access times within that band. Tipton
(2015) argues that the introduction of Container Tables was
The Sleuth Kit. The Sleuth Kit (TSK) is an open-source filesystem important to treat multi-tiered storage systems more efficiently. In
forensic tool. It was developed with high portability and extensi- a multi-tiered storage system, it is common to have different types
bility in mind (Altheide and Carvey, 2011, p. 42). The highly of storage media that offer different reading and writing charac-
extensible design of TSK allows developers to extend its function- teristics. Some storage media such as flash memory allow per-
ality as well as the types of file systems it supports. TSK is struc- forming random access faster than traditional hard disks, which are
tured into multiple layers of abstraction that map how data is more suitable to perform sequential operations.
stored on storage media. The File System Layer is of most impor- In a multi-tiered storage system, data needs to be reorganized
tance within this work, as this is the abstraction where file systems according to its characteristics. To perform this reorganization, it is
such as FAT, NTFS and also ReFS are settled. possible to swap bands, and thus their contents between different
Copy-On-Write. Copy-On-Write (COW) is an update policy that storage tiers. The additional tracking of metadata information
may be used to alter data on a storage medium. A COW update within a band allows to monitor its heat, so to say how often data in
policy makes sure that data is never updated in place. Whenever it is accessed. This performance metric may be used to decide when
the content of a block should be altered, the block is read into to shuffle two bands.
memory, modified and its content is written to an alternate location This concept as it is described by Tipton (2015) and Das (2017) can
on the storage medium. When the system crashes while a block is be found in current ReFS versions. It is important to note that
updated through a COW policy, the old state of the data remains shuffling two different bands also changes the respective physical
untouched and still persists. COW offers a simple strategy to addresses of the data found in the bands. Before such a shuffle
enforce atomicity in write operations and to ensure the integrity of operation it would be necessary to adjust all pointers that reference
data structures (Rodeh et al., 2013, p. 15). data in the affected bands. To prevent the necessity of updating any
pointers, ReFS v2 started implementing virtual addresses as a
3. ReFS internal structure mechanism to leverage shuffle operations within containers. Nearly

We analyzed the structure of ReFS and classified identfied Table 1

structures into the categories File System, Content, File Name, Met- Tables referenced by the checkpoint structure.
adata and Application introduced by Carrier (2005). Detailed
Table Identifier Table Name
structure description of ReFS which are important for forensic an-
alyses can be found in our technical report (Prade et al. (2019)). In 0x2 Object ID Table
0x21 Medium Allocator Table
this work we describe only the structures important for enumer- 0x20 Container Allocator Table
ating and recovering files and folders. The ordering corresponds to 0x1 Schema Table
the occurrence when starting interpreting an ReFS partition at the 0x3 Parent Child Table
boot sector. 0x4 Object ID Table, duplicate
0x5 Block Reference Count Table
0xb Container Table
3.1. Checkpoint 0xc Container Table, duplicate
0x6 Schema Table, duplicate
ReFS partitions have 2 checkpoint structures which are written 0xe Container Index Table
alternately, to have at least one valid checkpoint in case of a crash. 0xf Integrity State Table
0x22 Small Allocator Table
The checkpoint of a file system is the first structure that holds the
 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915 S3

all addresses used under ReFS have to be translated into real ad- While Copy-On-Write is used to leverage atomic write operations,
dresses before they may be used. After two containers were shuffled, duplicate table structures seem to be used to battle bit rot. If one
it is only necessary to update their respective address mappings. variant of the Object ID Table becomes corrupted, the ReFS driver
The Superblock, the Checkpoint, the Container Table, and the may fall back to using the other variant of it.
Container Allocator Table all use real addresses that do not need to
be translated first, because they are needed to bootstrap address
translation. We did not examine the inner structure of the rows of 3.4. Directory tables
the Container Table in more detail. As of now, only two fields in
these rows are known to us. One of these fields is required to Directory Tables implement the logic of directories. Every
perform the address translation process. Directory Table contains a single row that stores metadata of the
Fig. 1 portrays the practical usage of the Container Table in the directory that the table represents. We refer to this row as the
address translation process. Every virtual address used under ReFS directory descriptor. The other rows found in a Directory
may be separated into a container- and an offset-component. The Table mostly represent directory entries that are contained in the
container component determines which row of the Container directory. There exist two different types of directory entries: file
Table needs to be queried to perform an address translation. The entry and directory link entry. A directory link mainly maps a name
offset component provides an offset that must be added to the to a directory ID, which can be lookup in the Object ID Table. For all
translated address. files in a Directory Table, an additional entry type exists, called ID2
that provides a mapping between the metadata address of a file and
its current handle.
3.3. Object ID table Metadata addresses are used to uniquely address files and di-
rectories. They offer a shorter notation than the path of a file. A
When conducting a forensic analysis of a ReFS formatted file metadata address in ReFS consists of two components, the direc-
system, the table of most importance is the so-called Object ID tory ID and the file ID which get concatenated to a 128-Bit metadata
Table. This table references the root nodes of a variety of other ta- address. The directory ID part identifies the Directory Table and
bles and associates an identifier to them. Alongside a table that the file ID the entry in this table. The directory identifier of a
contains general information about the volume, a table storing an directory is equal to its table identifier. A table with the id 0x800
Upcase Table and a table containing an entry point to the redo log, represents the directory with the metadata address 0x800|0. The
the Object ID Table also references the root nodes of all Directory files inside this directory are referred to as 0x800|i, where i > 0
Tables. When searching for any directory, the Object ID Table must and i < max_file_id0x800. This choice of addressing files induces a
be queried. After a directory has been deleted, it is not referenced tight coupling between metadata addresses and the actual paths in
by the Object ID Table anymore. The Object ID Table is the only which files reside.
place where the actual addresses of Directory Tables are stored. If a file is moved from one directory to another directory its
Because of these circumstances, recovery techniques that attempt metadata address is altered as a new directory identifier, and a new
to restore directories under ReFS should focus on recovering rows file identifier are assigned to it. However, it is still possible to refer
found in the Object ID Table. to the file by its original file- and directory-identifier. The original
The Object ID Table additionally stores meta-information about identifiers are still saved in a field of the metadata entry of the file.
the tables that it references. It stores the addresses and the Additionally, an ID2 row in its original Directory Table gets created.
checksums of the tables it references, as well as their last persisted This row is used to offer a mapping between the original- and the
log sequence number in the file system journal. Additionally, en- current-metadata address of the file. This measure allows metadata
tries in the Object ID Table may also store a buffer with variable addresses of files to be stable even if a file is moved into a new
data. For links to Directory Tables, this buffer is filled with the next directory.
file identifier/metadata address to be used in the directory. There exist two special Directory Tables. The root directory (ID
The root Directory Table is stored with the ID 0x600. Any other 0x600) and the File System Metadata directory (ID 0x520)
regular Directory Table get an ID assigned which is greater then which fulfills the similar purpose as the $Extend directory known
0x700. from NTFS.
As the Object ID Table takes a superior role to the tables which it To locate an entry by its metadata address, one has to search the
references, more importance is attached to guarantee its integrity. Directory Table in the Object ID Table with the directory ID part.
A so-called Duplicate Object ID Table exists which contains the same After that, one has to locate the ID2 entry with the file ID, which
entries as well as the same slack space as the regular Object ID Table. links to the file entry.

Fig. 1. Exemplary address translation process.

S4 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915

3.5. File table We experienced that this non-referenced data is also copied by
the COW strategy. We call this slack space propagation and verified
Root nodes of File Tables are embedded within Directory Tables. this in an experiment where we filled slack space of a leaf node
Like all other tables, they start with a fixed-sized chunk of data that with markers. During usage of the filesystem (FS), these markers
is used to describe table-specific information. The entries stored in get propagated to new copies of this node, when data is written in a
a File Table are the properties that a file possesses. This concept COW manner.
strongly reminds of attributes that are used in NTFS. In this manner, We can use the concept of non-referenced data in nodes as well
MFT entries that are known from NTFS expose a similar behavior as as old non-referenced nodes to recover data in ReFS partitions. For
file-descriptor-tables in ReFS. Both consist of a small fixed recovering entries from nodes, we have to scan the entire data area
component as well as a variable-sized list of attributes. for indicators. An entry starts with a header which we can check for
We found the following attributes to be practically used by the plausibility.
ReFS driver: $DIR_LINK, $INDEX_ROOT, $DATA, $NAMED_DATA, As a second method, we can search the entire partition for tree
and $REPARSE_POINT. Most of the information that was previ- node pages. Every node and even every page in ReFS start with a
ously stored in the $STANDARD_INFORMATION attribute of NTFS specific header. This header offers a great opportunity for identi-
has now become a part of the fixed data found in file descriptors. fying and locating all nodes in ReFS.
Since the form in which contents of a directory are represented was
utterly changed and shifted into the responsibility of rows in 5. Implementation
Directory Tables, both the $INDEX_ROOT and the $INDEX_-
ALLOCATION attribute known from NTFS seem to have become 5.1. TSK extension
obsolete. Still, we found all directories to use equally filled dummy
$INDEX_ROOT attributes. With the knowledge we gained through analyzing ReFS we
Aside from the $DATA attribute, the contents of all attributes under implemented an extension for TSK to support interpreting ReFS.
ReFS seem to be stored resident, meaning in place. In NTFS it was ReFS shares the same DOS partition type as exFAT and NTFS, so we
possible for the $DATA attribute to either be stored resident or non- modified the mmls tool which analyses partition tables to output
resident. In ReFS, the $DATA attribute now seems always to be stored ”ReFS” additional to exFat and NTFS as the output of filesystem
non-resident. Even if a file is only a few bytes large, ReFS allocates an names.
own cluster for it and saves its data non-resident. Furthermore, the The other implementations focus on the file system layer. TSK
$DATA attribute seems to be the only attribute that spans an embedded uses a list of filesystem openers which can be used to try opening
tree, which stores rows of cluster runs. This approach also makes it easy filesystems without knowing the type beforehand. The opener
to search for an offset relative in the file as the tree used to store data returns a context object which stores basic information about an
runs is collated by the relative start of a data run. filesystem such as block size, first/last block of the FS and first/last
metadata addresses. Additionally, it provides the TSK API as func-
4. ReFS data recovery tion pointers which allows to interact with the FS in a generic way.
Our opener implementation first parses and checks (with
In ReFS, most data are organized in key-value stores, so called included signatures and checksums) the boot sector by reading the
tables. Internally, these tables are implemented as a Bþ -tree. first sector of the FS. Additionally, we get the cluster and sector size
Microsoft calls their implementation Minstore Bþ. from the boot sector.
When data gets written in ReFS, the updates are not performed After identifying a valid ReFS boot sector, we parse the super-
in-place, instead a COW strategy is used. In the Bþ -tree, the payload block which resides in cluster 30 as well as the superblock backups
is always held in the leaf nodes. When data gets altered in this table at the end of the FS and pick the first valid one. We check the su-
structures, a new leaf node is created which takes over the old data perblock with its self-checksum. From the superblock we get the
and applies the modifications. Afterwards the pointer in the parent volume signature and the cluster addresses of the checkpoint
node has to be adjusted, which is also done with the COW strategy. structures.
This process bubbles up to the root node. This principle is shown in
Fig. 2 where pointer adjustments bubbles up and create a new root
node, by inserting 19 into the leaf node.
When an entry is removed from a node of the Bþ -tree, the entry
is not wiped. Instead, only a link to a data chunk is removed and the
data chunk is released. Fig. 3 shows a node with deleted entries e1 ,
e2 and e4 . They stay untouched, only the key indexes are made
invalid. On the right-hand side the entry e5 gets inserted which
overwrites big parts of e2 . Every time a entry is inserted or deleted,
the key index gets reordered. Fig. 3. Data organization in leaf nodes.

Fig. 2. (a) A basic b-tree (b) Inserting key 19, and creating a path of modified pages (based on Rodeh et al. (2013)).
 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915 S5

Fig. 4. Combining entries of existing and deleted tables that are referenced by the Object ID Table.

Our implementation picks the most recent valid checkpoint implemented an refs_fsstat function which outputs informa-
after parsing both checkpoints. Hitherto we refrain from combining tion from the Volume Information Table. A brief overview of the
both checkpoints because it is difficult to combine these tree other context functions implementations is given next.
structures, although one checkpoint references older root nodes inode lookup. This function is used to get general metadata for a
left-behind through the COW process. 13 root node references get file or directory specified by the provided metadata address. Every
extracted from the chosen checkpoint including e.g. Container, metadata address under ReFS is formed by a file identifier and a
Object ID and Volume Information Tables. directory identifier. To find the location of a file, one must query the
Next the Container Table gets enumerated and address trans- directory table which is identified by the directory identifier. With
lation data is extracted which is later used to translate page refer- the file identifier as a key, the directory table gets queried and the
ences to physical page addresses. found directory descriptor- or file table can then be parsed. The
One of the main parts is the enumeration of the Object ID data attribute is non-resident as the data of files in ReFS resides in
Table as it references directories among others. It should not matter external clusters. The implementation enumerates the data table to
whether the regular Object ID Table or its identical copy, the receive all data runs. An enumeration strategy function attempts to
duplicate Object ID Table is processed since both should store the recover data runs that have been removed from the index. After-
same contents and presumably also contain the same slack space. wards, the data is transformed to the TSK metadata representation.
Many tables referenced by the Object ID Table represent directories. dir open meta. This function associates the names of files and
Because of that, it is not only important to read all regular entries folders to their metadata addresses. This function needs the met-
found in the Object ID Table but also to recover deleted ones, which adata address of a folder to process. It populates a directory data
can be identified by signatures in their key. structure with mappings of file names and associated metadata
When recovering links to Directory Tables, it is possible to find addresses. For this the whole directory table gets enumerated and
multiple links with the same table identifier. This behavior mainly the file and directory links are transformed and inserted into the
occurs as a result of the COW process, that continually writes the directory data structure.
root nodes of tables to new pages. Duplicate page references that file get sidstr. For our implementation we reused large parts of
store the same table identifier as others may reference different the NTFS implementation. For a given metadata entry it extracts its
root nodes. As a consequence of that, it is possible for a directory to security identifier (SID) string.
be formed by multiple tree structures which store older states of
that folder. An example of this state of affairs is shown in Fig. 4. The
grey marked rows in the figure represent rows that were recovered
in the Object ID Table. When combining the keys found in the
different restored tree structures, it is important that all keys must
only be interpreted once. The current state of the table stores the
keys 2, 4, 6, and 9. These are interpreted as regular entries. The
recovered trees must be sorted in descending order of their recency
represented with the virtual allocator clock. After a tree has been
enumerated completely, all new keys are added to the set of exis-
tent keys. Keys that are not found in the regular referenced tree
structure (3, 7) but are located in different tables that were
recovered, are considered to be deleted entries. It is only attempted
to recover deleted entries which reference Directory Tables. All
other table structures referenced by the Object ID Table are regu-
larly processed by querying its index.
The TSK tool fsstat outputs general filesystem information. It
depends on a fsstat function provided from the context object. We Fig. 5. Classification process in the collection phase of the carver.
S6 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915

Usnjls. ReFS allows for creating an Update Sequence Number which starting address they refer. Fig. 6 illustrates this issue. The
(USN) journal similar to NTFS. The implementation of this tool can red marked file system starts at the beginning of the disk image.
display the content of an USN journal which is stored as a common Address references used in this file system may be interpreted
file. We extended the NTFS implementation with a new USN record without adding any offsets. The blue marked file system starts
format present in ReFS USN journals. somewhere in the middle of the disk image. Therefore, its starting
One major drawback in our implementation is, that TSK sup- offset relative to the start of the disk image needs to be added to all
ports only 64-bit large metadata addresses to reference files and cluster references that are interpreted in it. Luckily, the cluster
directories. ReFS however uses two 64-bit identifier to refer to a numbers found in the page header of the Container Table are equal
metadata entry. One encodes the directory table in which the to their physical addresses. Thus, the first found page that belongs
metadata entry resides. The other is used to address the entry to a container table may be used to determine the starting location
within the directory table. Therefore, we changed the definition of of a ReFS file system.
the TSK_INUM_T from uint64_t to a struct that holds a low and a The collection process may also be halted at any time. Whenever
high 64-bit value. This change however makes all application using the carver has read and classified 256 MiB of data, it writes its
TSK incompatible to our implementation and need to get revised in progress state as well as the list of found pages into a file. When
the future. restarting the application at a later time, the collection process may
Another drawback is, that the COW creates multiple copies of continue at the last written state. The idea of this feature as well as
files which represent their state at different times. TSK does not lots of other concepts, were extracted from the refsutil.exe
cover the idea of multiple copies of a file with a shared metadata utility. refsutil.exe was introduced in early 2019 by Microsoft.
address. Major changes have to be made in TSK to support this It allows to verify or recreate a boot sector, to reclaim leaked
concept in the future. clusters, to remove corrupt files, or to recover files from a ReFS file
system. The recovery operation is also referred to as salvage. We
5.2. Page carver analyzed the implementation of this operation to get valuable for
recovering files.
Even though many deleted entries may be restored by analyzing Reconstruction phase. After all pages have been read and classi-
remnants in tree structures, still many nodes of previously allo- fied according to Fig. 5, the carver goes into the reconstruction
cated pages remain unseen. The reason for this is that some nodes phase. The reconstruction phase loops over all found volume states
are not referenced by any accessible page anymore and there exists and performs identical operations on them. First, the latest state of
no way to locate them. It is possible that at an arbitrary position on the Container Table is restored so that it is possible to translate
the disk, important data is located that belongs to the metadata of a virtual addresses in a volume to real addresses. Next, the carver
deleted file or a directory structure. If we, however, have no way to loops over all directory tables found in a volume.
find this structure, it remains unseen and potentially substantial Metadata reconstruction. Directory tables are stored as a flat
evidence is suppressed. The implementations for the file systems in sequence of pages. The carver searches these pages for signatures
TSK are purely based on parsing existing structures and thus react that may be used to identify files, directory links and directory
highly responsive. It would, however, be necessary to scan an entire descriptors. When a corresponding signature is found, the carver
volume if all deleted pages have to be recovered. Therefore, a carver executes various sanity checks on the found data and tries to
independent of the actual ReFS implementation was developed. interpret it as the structure it believes it to be. The carver must also
The carver is based on the tsk_img_open function provided by restore the subtree formed by directory descriptor tables and file
TSK. A user may provide an arbitrary image and an offset to the tables to access their attributes, and their data runs. If the carver
carver which then tries to recover deleted files and directories from was successful in restoring an entry, the entry is saved in the class
the image. The carver works in two steps. format file_information or dirref_information. Every directory
Collection phase. This is the phase in which the carver starts. The stores these entries as a set.
carver scans the entire provided image for blocks that look like If a file_information entry or a dirref_information entry
pages of tree nodes. The step size of the carver is 4 KiB. In every is equal to an already existing entry, it is discarded. To determine
step, it verifies the page header within the first 80 bytes of a read whether two entries are identical, multiple properties of them are
block. If these bytes fulfill the characteristic properties of a page compared. It would not be sufficient to merely check whether the
header, the page is kept for further analysis. If the table identifier of file identifier already exists in the set of found entries as this would
the page is equal to the table identifier of a container table or any discard potential older copies of files. Instead, the file identifier, the
directory table the page is retained, else it is discarded. last modification time, and the file size are used in conjunction to
The carver maintains a structure that maps volume signatures to check whether two file entries are equal. If they are not equal, both
an object that represents the state of a ReFS file system. This may be contained in the same result set.
structure is used to distinguish different instances of ReFS file Extraction phase. After all directory pages of a volume have been
systems that may exist if the underlying disk contains multiple analyzed and their entries have been transformed into the internal
ReFS partitions or was formatted multiple times with the ReFS file representation of files and directories, the extraction phase begins.
system. Within the state of each volume, a map of directory iden- First all directory tables found in the volume are created as folders
tifiers is maintained. This map assigns table identifiers to a list of of the form <vol_sig>/<dir_id>. Next, a volume report with
pages. When looking up the key to a directory table in this map the the file name <vol_sig>/report.txt is created. The report
caller may obtain a list of all pages that store information relevant describes the properties of the corresponding volume such as its
to this table, so to say all nodes that once belonged and nodes that
still belong to a table with a given identifier.
The carver also derives further information from pages that
were read. The number of addresses stored in the page header is
used to decide whether the page belongs to a volume with a cluster
size of 4 KiB or 64 KiB. It is also important for the carver to know at
which disk offset the volume started. At a later time, the application
must interpret cluster runs. For this, it is necessary to know to Fig. 6. Multiple file systems at different locations within a disk.
 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915 S7

offset in the disk image, its cluster size and the number of di- generate a more “realistic” reflection of the usage of a filesystem.
rectories that it contains. The report also describes the address The distribution of the actions, however, was an arbitrary guess of
translation mapping that was extracted from the container table of the authors of how often these actions occur.
the volume. After every action we documented the outcome in a modified
Next, the application creates a file called <vol_sig>/struc- TSK body file.2 Both, the TSK implementation as well as the carver
tures.txt that is filled with the reconstructed directory hierarchy were adjusted so that they were able to output the modified body
formed by the file system. Since the application stores the extracted file format to compare results. The following characteristics were
directories in a flat form and only refers to them through their stored in the modified body file for every file or directory present
directory identifiers, this file may be used to re-establish directory on the filesystem: An MD5 checksum of the file content, which is
structures. Finally, the application iterates over all directory struc- set to 0 for directories. The name of a file or a directory without its
tures and dumps the files found in them. Into every directory a file complete path. The current directory identifier of a file, for di-
called <vol_sig>/<dir_id>/directory_report.txt is rectories their parent identifier is stored. The initial metadata
written which contains a list of all files found in the directory. The address of the file or folder, which stays constant, even if the file
list also contains the metadata of duplicate files that may have been gets moved. The MAC timestamps which track the last modifica-
created through the Copy-On-Write mechanism. The contents of all tion- access- and the creation-time of a file. The size of a file.
files are finally dumped into the corresponding directory. To pre- The python applications that are used to acquire the ground
vent naming conflicts among restored files that utilize identical file truth of the state of the file system use the function os.stat,
names files are written as <vol_sig>/<dir_id>/<fil- which was only able to obtain a 64-Bit metadata address. For our
e_id>_<copy_id>. Their names however may easily be looked evaluation this was no limitation because all created metadata
up by examining the <vol_sig>/structures.txt or the addresses were small enough.
<vol_sig>/<dir_id>/directory_report.txt file. The capture of the final state of the file system includes all
As of now, the carver is only able to extract files and directories existent files and directories, whereas the outcomes of the single
of a file system, and contrary to the ReFS implementation in the file executed operations only log changes made to files and the creation
system layer of TSK does not show additional information about a of directories. Every action that creates, alters, moves or copies a file
volume. It might be best to use both of those tools in conjunction. generates a log entry for the modified state of the file. We refrained
The file system implementation of TSK provides valuable infor- from logging the changes in the metadata of directories resulting
mation such as attributes of files as well as file system metadata from file actions as a deliberate decision in weighting the results. In
that has not yet been added to the analysis process of the carver. A practice, it might be of more importance restoring metadata entries
crucial advantage of the implementation in the file system layer of of files than of directories. We did not want to weight the incapa-
TSK is that its implementation can decide whether a file is allocated bility to restore an old timestamp of a directory equal to the inca-
or has been removed. The carver is not aware of whether the files pability of restoring the contents of a file.
that are extracted were allocated or not. On the other hand, the
carver provides a valuable tool for an investigator that intends to
6.1. TSK extension
examine a corrupted ReFS file system or an investigator who wants
to obtain potential older copies of files.
It is expected that the TSK extension can produce a report equal
to the actual state of the file system. We tested the TSK extension on
6. Evaluation
the 8 generated images and called the tool fls with the option to
only display allocated files. The resulting output should be equal to
As of now, there exists no work that looks at recovering files and
the current state of the file system that Windows reports.
folders from ReFS formatted file systems. Thus also, no work has yet
As shown in Table 2, for all randomly generated testing sce-
stated test cases that could be used to evaluate tools that serve such
narios, the output of the TSK extension matches the actual state of
a purpose. We designed multiple test scenarios which portrayed a
the file system, except the folder “System Volume Information”
fictive usage of the file system. The evaluation process of the tools is
(Table ID: 0x701) and its content “WPSettings.dat” and “Index-
strongly based on the description of the evaluation dataset used by
erVolumeGuid”. The numbers x=y count the number of detected
Plum and Dewald (2018). We generated 8 ReFS images with
files/folders (x) and the number of files/folders present on the final
different configurations of overall size (2GiB, 5GiB, 40GiB, 100GiB)
state (y). These entries were not included in the extracted final state
and cluster size (4KiB, 64KiB). On each we performed 1000 actions
of the file system because Windows hides this folder from regular
from the following list with different likelihood of occurrence:
users. The same applies to the “File System Metadata” (Table ID:
0x520) folder, which we omitted proactively in this output of the

add file PðXÞ ¼ 0:35: Add a random file from the EDRM File
TSK, since it is not a regular directory. The results show that we are
Formats Data Set 1.01 to the volume
able to output the exact actual ReFS state with our TSK extension.

delete file PðXÞ ¼ 0:2: Delete a random file

move file PðXÞ ¼ 0:05: Moves a random file

change file PðXÞ ¼ 0:1: Insert a random amount of A's into a 6.2. File recovery capabilities
random file

copy file PðXÞ ¼ 0:1: Copies a random file With this evaluation we want to compare the capability of

add folder PðXÞ ¼ 0:15: Creates a randomly named folder recovering deleted files and reconstructing previous states of files

delete folder PðXÞ ¼ 0:05: Removes a randomly picked folder and folders. We compare the outputs of the developed tools with
the action trace log which was logged during test image creation.
If an action could not be executed in a step (e.g. removing a In the tables referenced in the following, the numbers x=y have
directory, if no directory exists at all), a different action was picked to be interpreted as the number of exact outputted files/folders (x)
randomly. This choice of the distribution was picked deliberately to and the number of all different file folder states once present on the

1 2
https://www.edrm.net/resources/data-sets/edrm-file-format-data-set/. https://wiki.sleuthkit.org/index.php?title¼Body_file.
S8 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915

filesystem (y). Table 3 compares the output of the TSK extension Table 3
only considering allocated files and folders to all states. Note- State of all allocated files (TSK extension), compared to action log.

worthy, less directories match than in the evaluation seen in Configuration Interpreted entries
Table 2. That is because we use the action log as comparison instead Directories Files
of the final filesystem state. In the action trace we only log directory
2GiB 4KiB 14/154 (9.09%) 110/613 (17.94%)
creation and no timestamp modifications due to changes of child
2GiB 64KiB 16/159 (10.06%) 81/590 (13.73%)
files and folders. Therefore, only few folders outputted by the 5GiB 4KiB 11/144 (7.64%) 121/606 (19.97%)
implemented applications which got never an update in time- 5GiB 64KiB 9/159 (5.66%) 31/585 (5.30%)
stamps match exactly the logged folders. As expected, the appli- 40GiB 4KiB 9/129 (6.98%) 76/615 (12.36%)
40GiB 64KiB 7/159 (4.40%) 141/616 (22.89%)
cations output at least all files present in the final image state.
100GiB 4KiB 6/147 (4.08%) 62/592 (10.47%)
Table 4 includes the file and directory recovery capabilities of 100GiB 64KiB 12/152 (7.89%) 108/593 (18.21%)
the TSK extension. Compared to table 3 more files and folders are
outputted correctly. As the ReFS extension can only restore entries
from referenced pages, it is likely that there remain pages that the
Table 4
TSK extension cannot locate. Additionally, the TSK extension only State of all allocated and recovered files (TSK extension), compared to action log.
retrieves the most recent state of a file and is unable to address and
Configuration Restored entries
thus to retrieve past states of a modified file.
Table 5 shows all files that could be recovered by using the Directories Files
carver. In all scenarios the carver is able to recover more files and 2GiB 4KiB 15/154 (9.74%) 132/613 (21.53%)
directories than the TSK extension. The major drawback of the 2GiB 64KiB 19/159 (11.95%) 125/590 (21.19%)
carver is, that it is unable to differentiate between existing and 5GiB 4KiB 14/144 (9.72%) 159/606 (26.24%)
5GiB 64KiB 28/159 (17.61%) 75/585 (12.82%)
removed files. 40GiB 4KiB 12/129 (9.30%) 97/615 (15.77%)
Table 6 conclusively give an overview of the runtimes of the 40GiB 64KiB 11/159 (6.92%) 175/616 (28.41%)
various test scenarios. To conduct all experiments, we used a 4 TiB 100GiB 4KiB 10/147 (6.80%) 113/592 (19.09%)
large Western Digital hard drive (Product ID: WDBHDW0040BBK). 100GiB 64KiB 13/152 (8.55%) 142/593 (23.95%)
With the application dd we estimated a sequentially read speed of
95e110 MiB/s. Depending on the size of a volume, the carver is
most of the time busy with reading and collecting pages from the Table 5
volume. The runtime of the TSK extension also varies strongly State of all allocated and recovered files (carver), compared to action log.

based on the number of pages that are read. Configuration Restored entries

Directories Files
6.3. Recovering old file states 2GiB 4KiB 15/154 (9.74%) 139/613 (22.68%)
2GiB 64KiB 21/159 (13.21%) 133/590 (22.54%)
In this evaluation we look at how the COW process impacts the 5GiB 4KiB 16/144 (11.11%) 169/606 (27.89%)
5GiB 64KiB 34/159 (21.38%) 88/585 (15.04%)
recovery of old file states. For this experiment we developed a small 40GiB 4KiB 16/129 (12.40%) 102/615 (16.59%)
application that writes text into a file in a ReFS file system. The text 40GiB 64KiB 11/159 (6.92%) 183/616 (29.71%)
was artificially written at the speed at which a person types (175 100GiB 4KiB 10/147 (6.80%) 124/592 (20.95%)
characters per minute). Every 2 min the text file was saved, and a 100GiB 64KiB 13/152 (8.55%) 150/593 (25.30%)
checksum of its current intermediate state was logged together
with its metadata.
Because of the COW policy that ReFS uses, it is likely that met- Table 6
adata, that describes the file and the location of its clusters, is Runtimes of the applications.

dispensed into multiple places of the volume. We used the devel- Configuration TSK TSK Carver
oped carver to find as many existent old copies of the file as (alloc. files) (all files)
possible. Additionally, we instrumented the ReFS driver to log 2GiB 4KiB 1.233 s 1.379 s 20.411 s
cluster allocation and COW information, e.g. when data is written 2GiB 64KiB 1.21 s 1.947 s 20.519 s
from one page to another. The data generated in this process 5GiB 4KiB 0.613 s 0.694 s 50.1 s
5GiB 64KiB 0.917 s 1.206 s 50.764 s
allowed us to reconstruct which intermediate states of the file were
40GiB 4KiB 0.972 s 0.94 s 383.027 s
generated. 40GiB 64KiB 2.256 s 2.409 s 402.892 s
The continuous modification of a single page in the COW pro- 100GiB 4KiB 1.441 s 1.415 s 997.922 s
cess can be viewed as a chain of page states. Every page state has a 100GiB 64KiB 2.037 s 2.344 s 1012.462 s

Table 2
Output of the TSK extension, compared to the final state of the file system.
physical address associated to it. If this physical address is reused at
a later time for an allocation, this state cannot be recovered
Configuration Interpreted entries anymore. All older pages in such a page chain that get not reallo-
Directories Files cated may still yield valuable contents.
2GiB 4KiB 47/46 (102.17%) 112/110 (101.82%) The experiment focused on writing data into a single file.
2GiB 64KiB 44/43 (102.33%) 83/81 (102.47%) Additionally, no other modifications on the file system were made
5GiB 4KiB 45/44 (102.27%) 123/121 (101.65%) while the file was altered. While this is a lab setup, it gives an upper
5GiB 64KiB 20/19 (105.26%) 33/31 (106.45%)
boundary for the recovery capability of old file states on ReFS
40GiB 4KiB 25/24 (104.17%) 78/76 (102.63%)
40GiB 64KiB 56/55 (101.82%) 143/141 (101.42%)
partitions.
100GiB 4KiB 35/34 (102.94%) 64/62 (103.23%) The experiment was conducted with five different data sets.
100GiB 64KiB 53/52 (101.92%) 110/108 (101.85%) Each data set was a 5 GiB large ReFS volume with a cluster size of 4
 P. Prade et al. / Forensic Science International: Digital Investigation 32 (2020) 300915 S9

Table 7 with page carving we can recover more data than with only using
Experiment to analyze the recoverability of COW copies. deleted entries which are still present on the disk.
Duration Save Op. Recovered Recoverable

States Pages Acknowledgements

10 m 5 3 (2 valid) 6/15
30 m 15 3 (3 valid) 5/31 This work was supported by the German Federal Ministry of
60 m 30 3 (3 valid) 5/47 Education and Research (BMBF) as part of the FIDI project.
120 m 60 3 (3 valid) 5/86
240 m 120 3 (3 valid) 4/141
References

Altheide, C., Carvey, H., 2011. Digital Forensics with Open Source Tools. Elsevier.
Carrier, B.. The Sleuth Kit. URL. https://www.sleuthkit.org/sleuthkit/. last visited:
KiB. For every data set, the text file was written for a different time 2019-10-04.
period, ranging from 10 min to 2 h. Carrier, B., 2005. File System Forensic Analysis. Addison-Wesley Professional.
Das, R., 2017. ReFS support for SMR drives. Presentation. SDC, 2017. https://www.
As seen in Table 7, the number of recoverable file states does not
snia.org/sites/default/files/SDC/2017/presentations/smr/Das_Rajsekhar_ReFS_
differ much between the various experiments. In all scenarios, the Support_For_Shingled_Magnetic_Recording_Drives.pdf.
valid files that could be recovered corresponded to the last 3 states Georges, H., 2018. Resilient Filesystem. Master’s thesis. NTNU. https://ntnuopen.
of the file, with the exception of the shortest run. That outlier oc- ntnu.no/ntnu-xmlui/handle/11250/2502565.
Green, P., 2013. Resilient File System (ReFS), Analysis of the File System Found on
curs because the carver found an empty version of the new created Windows Server 2012. Staffordshire University. Technical Report.
file. In the rightmost column of Table 7 you can see that even Metz, J., 2013. Resilient file system (ReFS). last visited: 2019-10-04. https://github.
though various runtimes were used for the experiments, and for com/libyal/libfsrefs/blob/master/documentation/ResilientFileSystem(ReFS).pdf.
Nordvik, R., Georges, H., Toolan, F., Axelsson, S., 2019. Reverse engineering of ReFS.
longer running experiments more page states were created, the Digit. Invest. 30, 127e147. https://doi.org/10.1016/j.diin.2019.07.004. http://
number of recoverable pages stays nearly constant. The right www.sciencedirect.com/science/article/pii/S1742287619301252.
number shows the total amount of allocated pages during the Plum, J., Dewald, A., 2018. Forensic APFS file recovery. In: Proceedings of the 13th
International Conference on Availability, Reliability and Security. ACM, New
runtime. It seems like the allocator in the ReFS driver reused old York, NY, USA, p. 47. https://doi.org/10.1145/3230833.3232808, 1e47:10. URL.
pages relatively fast. https://dl.acm.org/citation.cfm?id¼3232808.
Prade, P., Grob, T., Dewald, A., 2019. Forensic Analysis of the Resilient File System
(ReFS) Version 3.4. Technical Report CS-2019-05. Department Informatik. URL.
7. Summary
https://opus4.kobv.de/opus4-fau/frontdoor/index/index/docId/12526,
10.2.32328085593/issn.2191-5008/CS-2019-05.
With this work we investigated the internal structures of the Rodeh, O., Bacik, J., Mason, C., 2013. BTRFS: the linux B-tree filesystem. ACM Trans.
Storage 9, 9. https://doi.org/10.1145/2501620.2501623. URL. https://dl.acm.org/
new Resiliant File System. We used the insights to extend The
citation.cfm?id¼2501623.
Sleuth Kit to be able to parse and interpret ReFS partitions. Open Sinofsky, S., 2012. Building the next generation file system for Windows: ReFS. URL.
that we implemented a page carver to recover file system data. We https://blogs.msdn.microsoft.com/b8/2012/01/16/building-the-next-
evaluated both tools and come to the conclusion that our TSK generation-file-system-for-windows-refs/. last visited: 2019-10-04.
Tipton, J., 2015. ReFS v2, Cloning, projecting, and moving data. Presentation. SDC,
extension works as intended and reports the current state of a ReFS 2015. https://www.snia.org/sites/default/files/SDC15_presentations/file_sys/
partition equal to the ReFS driver of Windows. We also showed that JRTipton_ReFS_v2.pdf.