Great question!

The example above shows how to read parquet through DataFrame which is part of Flow Framework.
To make it memory safe you need to get rid of ->collect() method (which is used here because there are only 4 rows and I wanted to display them as a table).

It's also important to remember that memory efficiency of parquet depends highly on how the file was created (what's the row group and page size), so it's not only about adjusting the reader (I'm in the middle of writing a blog post that explains it in details)

But if you just want ot read parquet file directly you can use flow-php/parquet library standalone, I didn't had time yet to prepare a proper documentation (shame on me) but you might be able to figure it out from tests: https://github.com/flow-php/flow/tree/1.x/src/lib/parquet/tests/Flow/Parquet/Tests/Integration/IO

I'm also writing a blog post explaining parquet and reading parquet so feel free to follow me on my social media for updates, or simply join our discord server if you have more questions (you can find the link in the top navigation of this website).

Thanks for asking!

Hi Norbert, thank you for the guidelines, happy to connect.

I tried the example below (taken from tests):

$reader = new Reader();
$file = $reader->read(__DIR__ . '/dataset/training.parquet');

foreach ($file->values(['text']) as $row) {
    var_dump($row);
}

But I get the error: "Class "Flow\Filesystem\SizeUnits" not found in C:\GitHub\neuron-llm\vendor\flow-php\parquet\src\Flow\Parquet\Reader.php:14"

I think the package depends on the filesystem package. Just to report to you and ask for confirmation.

Oh I must have miss that! Thanks for reporting, will check if it's missing dependency or incorrect reference between packages and will provide a proper fix, thank you 💪