Fully managed, safe, extremely fast .NET library to 📖read and ✍️write Apache Parquet files designed for .NET world (not a wrapper). Targets .NET 8, .NET 7, .NET 6.0, .NET Core 3.1, .NET Standard 2.1 and .NET Standard 2.0.

Whether you want to build apps for Linux, MacOS, Windows, iOS, Android, Tizen, Xbox, PS4, Raspberry Pi, Samsung TVs or much more, Parquet.Net has you covered.

• 0️⃣ Has zero dependencies - pure library that just works anywhere .NET works i.e. desktops, servers, phones, watches and so on.
• 🚀Really fast. Faster than Python and Java, and alternative C# implementations out there. It's often even faster than native C++ implementations.
• 🏠NET native. Designed to utilise .NET and made for .NET developers, not the other way around.
• ❤️‍🩹Not a "wrapper" that forces you to fit in. It's the other way around - forces parquet to fit into .NET.
• 🦄Unique Features:
  • The only library that supports dynamic schemas.
  • Supports all parquet types, encodings and compressions.
  • Fully supports C# class serialization, for all simple and complex Parquet types.
  • Provides low-level, high-level, and untyped API.
  • Access to file and column metadata.
  • Integration with DataFrames (Microsoft.Data.Analysis).

Parquet is designed to handle complex data in bulk. It's column-oriented meaning that data is physically stored in columns rather than rows. This is very important for big data systems if you want to process only a subset of columns - reading just the right columns is extremely efficient.

As a quick start, suppose we have the following data records we'd like to save to parquet:

1. Timestamp.
2. Event name.
3. Meter value.

Or, to translate it to C# terms, this can be expressed as the following class:

class Record {
    public DateTime Timestamp { get; set; }
    public string EventName { get; set; }
    public double MeterValue { get; set; }
}

Let's say you have around a million events like that to save to a .parquet file. There are three ways to do that with this library, starting from easiest to hardest.

The first one is the easiest to work with, and the most straightforward. Let's generate those million fake records:

var data = Enumerable.Range(0, 1_000_000).Select(i => new Record {
    Timestamp = DateTime.UtcNow.AddSeconds(i),
    EventName = i % 2 == 0 ? "on" : "off",
    MeterValue = i 
}).ToList();

Now, to write these to a file in say /mnt/storage/data.parquet you can use the following line of code:

await ParquetSerializer.SerializeAsync(data, "/mnt/storage/data.parquet");

That's pretty much it! You can customise many things in addition to the magical serialisation process, but if you are a really lazy person that will do just fine for today.

Another way to serialise data is to use Untyped serializer.

And finally, the lowest level API is the third method. This is the most performant, most Parquet-resembling way to work with data, but least intuitive and involves some knowledge of Parquet data structures.

First of all, you need schema. Always. Just like in row-based example, schema can be declared in the following way:

var schema = new ParquetSchema(
    new DataField<DateTime>("Timestamp"),
    new DataField<string>("EventName"),
    new DataField<double>("MeterValue"));

Then, data columns need to be prepared for writing. As parquet is column-based format, low level APIs expect that low level column slice of data. I'll just shut up and show you the code:

var column1 = new DataColumn(
    schema.DataFields[0],
    Enumerable.Range(0, 1_000_000).Select(i => DateTime.UtcNow.AddSeconds(i)).ToArray());

var column2 = new DataColumn(
    schema.DataFields[1],
    Enumerable.Range(0, 1_000_000).Select(i => i % 2 == 0 ? "on" : "off").ToArray());

var column3 = new DataColumn(
    schema.DataFields[2],
    Enumerable.Range(0, 1_000_000).Select(i => (double)i).ToArray());

Important thing to note here - columnX variables represent data in an entire column, all the values in that column independently from other columns. Values in other columns have the same order as well. So we have created three columns with data identical to the two examples above.

Time to write it down:

using(Stream fs = System.IO.File.OpenWrite("/mnt/storage/data.parquet")) {
    using(ParquetWriter writer = await ParquetWriter.CreateAsync(schema, fs)) {
        using(ParquetRowGroupWriter groupWriter = writer.CreateRowGroup()) {
            
            await groupWriter.WriteColumnAsync(column1);
            await groupWriter.WriteColumnAsync(column2);
            await groupWriter.WriteColumnAsync(column3);
        }
    }
}

What's going on?

1. We are creating output file stream. You can probably use one of the overloads in the next line though. This will be the receiver of parquet data. The stream needs to be writeable and seekable.
2. ParquetWriter is low-level class and is a root object to start writing from. It mostly performs coordination, check summing and enveloping of other data.
3. Row group is like a data partition inside the file. In this example we have just one, but you can create more if there are too many values that are hard to fit in computer memory.
4. Three calls to row group writer write out the columns. Note that those are performed sequentially, and in the same order as schema defines them.

Read more on writing here which also includes guides on writing nested types such as lists, maps, and structs.

Reading data also has three different approaches, so I'm going to unwrap them here in the same order as above.

Provided that you have written the data, or just have some external data with the same structure as above, you can read those by simply doing the following:

IList<Record> data = await ParquetSerializer.DeserializeAsync<Record>("/mnt/storage/data.parquet");

This will give us an array with one million class instances similar to this:

Of course class serialisation has more to it, and you can customise it further than that.

Read here for more information on how to read untyped data.

And with low level API the reading is even more flexible:

using(Stream fs = System.IO.File.OpenRead("/mnt/storage/data.parquet")) {
    using(ParquetReader reader = await ParquetReader.CreateAsync(fs)) {
        for(int i = 0; i < reader.RowGroupCount; i++) { 
            using(ParquetRowGroupReader rowGroupReader = reader.OpenRowGroupReader(i)) {

                foreach(DataField df in reader.Schema.GetDataFields()) {
                    DataColumn columnData = await rowGroupReader.ReadColumnAsync(df);

                    // do something to the column...
                }
            }
        }
    }
}

This is what's happening

1. Create read stream fs.
2. Create ParquetReader - root class for read operations.
3. The reader has RowGroupCount property which indicates how many row groups (like partitions) the file contains.
4. Explicitly open row group for reading.
5. Read each DataField from the row group, in the same order as it's declared in the schema.

You can also use web based reader app to test your files, which was created using this library!

If you have a choice, then the choice is easy - use Low Level API. They are the fastest and the most flexible. But what if you for some reason don't have a choice? Then think about this:

• Serialisation
• Writing
• Reading
• Diving deeper
  • Schema
  • Column
  • Nested types
  • Metadata
• Encodings
• Untyped serializer
• DataFrame
• Utilities
• Parquet getting started
• Contributing
• Special thanks

• Azure Cosmos DB Desktop Data Migration Tool.
• RavenDB - An ACID NoSQL Document Database.
• Cinchoo ETL: An ETL framework for .NET.
• ParquetViewer: Simple Windows desktop application for viewing & querying Apache Parquet files.
• ML.NET: Machine Learning for .NET.
• PSParquet: PowerShell Module for Parquet.
• Omni Loader: Self-tuning Database Migration Accelerator.
• Contoso Data Generator V2 : sample data generator.
• Recfuence - An analysis of YouTube's political influence through recommendations.
• Kusto-loco - C# KQL query engine with flexible I/O layers and visualization.
• DeltaIO - Delta Lake implementation in pure .NET.
• Personal Data Warehouse - Import(Excel/Parquet/SQL/Fabric)-Transform(C#/Python)-Report(SSRS).
• FastBCP - Export to parquet files in parallel from Oracle, SQL Server, MySQL, PostgreSQL, ODBC, Teradata, Netezza, SAP HANA, ClickHouse in one command line (Windows & Linux).

...raise a PR to appear here...

See the contribution page. The first important thing you can do is simply star ⭐ this project.

Without these tools development would be really painful.

• Visual Studio Community - free IDE from Microsoft. The best in class C# and C++ development tool. It's worth using Windows just because Visual Studio exists there.
• JetBrains Rider - for their cross-platform C# IDE, which has some great features.
• IntelliJ IDEA - the best Python, Scala and Java IDE.
• LINQPad - extremely powerful C# REPL with unique visualisation features, IL decompiler, expression tree visualiser, benchmarking, charting and so on. Again it's worth having Windows just for this tool. Please support the author and purchase it.
• Benchmarkdotnet - the best cross-platform tool that can microbenchmark C# code. This library is faster than native ones only thanks for this.
• You starring ⭐ this project!