Purpose

Enable storage of entire files intact, without having to read and serialize them, similar to attachments. Enable the use of these attachments through GUI drag-and-drop features. The attached files are not browsable outside DataJoint. DataJoint must maintain data integrity.

Operation

Attachments use a mechanism similar to external storage using file hashes as the filenames in a storage repository. The datatype should be either attach or attach-suffix where the suffix can be up to eight character long. The storage protocol and location (filesystem, S3, globus) is specified similar to external storage.

The table ~attach is created and maintained similar to how ~external is used for external storage. The only difference is that it also stores the original filename of the stored file.

Just as with external, this mechanism ensures that inserting the same file multiple times within the same schema does not result in duplicated storage.

Insert

Insert works by inserting the full path to the file to be inserted. The insert is asynchronous and commits the insert into the table into ~attach.
In its initial implementation, the entire file will be first read into memory for hashing and writing into storage.

Fetch

Fetch downloads the attached files. The download will be implemented asynchronously in the initial implementation. While the file is being downloaded, it is given the extension .download. When the download is complete the file is renamed into its target name.

The destination folder can be specified in the configuration. That can be overridden by the optional argument download_path of fetch. If neither is identified the download is performed into the current directory.

In all cases, the full path to the downloaded file is returned in the fetch output as attribute values.

Similar to external storage, fetch can be configured to use local caching.

If the same file retrieved multiple time as part of the same fetch, the file is only retrieved once and the same path in returned for each requested attribute value.

If the file already exists, the downloaded file is given an incrementing suffix.

Delete

Similar to external, delete only operates on the host tables and does not remove entries from ~attach or from the file repository.

Cleanup of ~attach and of storage folder work identically to the corresponding mechanisms of external storage.

Cleanup commands:

schema.attachment.delete_garbage()   # transaction-safe
schema.attachment.cleanup_store()   # not transaction-safe