We currently support three ways to im- and export data:

This interface was built with testing in mind. For operator tests, we need an easy, but more importantly reliable, way to load tables. We kept the code to a minimum so that we can be reasonably sure that any issues come from the tested operator, not some blown-up multi-threaded import component. Our tbl files look like this:

a|b
int|float_null
123|null
1234|457.7
12345|458.7

The first row gives the column names. This is followed by the column types and _null may be added for nullable columns. Finally, the values come and null (not NULL) is used for null values. We do not support quoting or escaping for strings and null becomes a null value in a string column.

To load this tbl file, simply call std::shared_ptr<Table> load_table(std::string file_name);.

We support csv files using the ImportCsv and ExportCsv classes. As these are regular operators, they can be placed anywhere in the execution plan for debugging purposes, similar to the Print operator. For importing, you will need to give the filename and an optional table name (otherwise, the base file name is used). Also, a configuration is needed, either in the form of CsvMeta or as a JSON file that is automatically loaded, e.g., mytable.csv.meta.

These settings include the column (default: ,) and row (\n) delimiters, the quote (") and escape (") characters as well as information about the column names and types. Finally, we have some options that are usually used in their strictest setting. These prevent quoted numbers from being accepted as well as rejecting null as a string.

Exporting is similar, only that the meta file is always generated.

CSV files have a couple drawbacks:

• They are written in a row-based fashion, which does not match our primarily column-bases data layout.
• They take a lot of space, which is a problem for big benchmark files.
• Because they length of rows is not clearly defined, importing takes more effort.
• As they are uncompressed, we need to recompress them, which takes some time as well.

Because the speed in which files can be loaded is relevant for how efficiently we can run benchmarks, we implemented our own binary format.

The ImportBinary operator takes a file and optionally a table name. The ExportBinary operator takes an input operator and a file name. We do not have any meta files for binary files, as everything is stored within. For a documentation of the format, see import_binary.hpp.