avdl

A Rust port of Apache Avro's avro-tools idl and idl2schemata commands. Compiles Avro IDL (.avdl) files to Avro Protocol JSON (.avpr) or Schema JSON (.avsc).

Usage

# .avdl → protocol JSON
avdl idl input.avdl output.avpr

# .avdl → individual schema files
avdl idl2schemata input.avdl outdir/

# stdin/stdout
avdl idl < input.avdl

# additional import search paths
avdl idl --import-dir ./extra/ input.avdl

Install

cargo install avdl

Or build and run from a local checkout:

cargo build
cargo run -- idl input.avdl

Testing

cargo test

The integration tests parse the same .avdl input files shipped in the Apache Avro test suite and compare output semantically against the expected .avpr/.avsc reference files. All test cases pass, alongside unit tests for serialization, error reporting, and edge cases. The tool is functional but young — bug reports are welcome.

Intentional divergences from Java

This tool aims for semantic correctness against the Java avro-tools output, but deliberately differs in a few ways:

• JSON formatting is not byte-identical. Whitespace and array line-breaking style may differ. JSON object keys are sorted alphabetically, whereas Java avro-tools preserves insertion order. The output parses to the same logical structure.

• Import search paths replace Java classpath. avro-tools resolves import paths via the JVM classpath; this tool uses explicit --import-dir flags instead, which serve the same purpose without requiring a JVM.

• Better error diagnostics. Errors include source context with the offending token underlined, powered by miette. The Rust tool also gives clear error messages in cases where avro-tools 1.12.1 crashes with unchecked exceptions (e.g., NoSuchElementException for duplicate message parameters, NullPointerException for reserved type names via backtick escapes).

  For example, given a record with a duplicate field name:

  record User {
      string name;
      int name;
  }
  

  avro-tools produces a Java stack trace:

  Exception in thread "main" org.apache.avro.SchemaParseException:
    org.apache.avro.AvroRuntimeException: Field already used: name type:STRING pos:0
      at org.apache.avro.idl.IdlReader.parse(IdlReader.java:220)
      ... 5 more
  

  avdl points directly at the problem:

  Error:   × parse IDL source
    ├─▶ parse `dup-field.avdl`
    ╰─▶ duplicate field 'name' in record 'User'
     ╭─[dup-field.avdl:4:13]
   3 │         string name;
   4 │         int name;
     ·             ──┬─
     ·               ╰── duplicate field 'name' in record 'User'
   5 │     }
     ╰────
  

• Nested unions are rejected per spec. The Avro specification states "Unions may not immediately contain other unions." This tool enforces this rule, while Java avro-tools 1.12.1 incorrectly accepts nested unions and silently produces an empty union [] in the JSON output. If you have .avdl files that compiled with Java but fail here, the nested union is a spec violation that Java should have caught.

• Float formatting uses serde_json defaults. Java renders large or small float/double values in scientific notation (e.g., -1.0E12); this tool uses serde_json's decimal representation (e.g., -1000000000000.0). Both parse to the same numeric value.

• Namespace validation covers all segments. Rust validates every dot-separated segment of namespace names. Java's IdlReader.namespace() loop skips the last segment, so a namespace like org.valid.0bad passes Java's validation but is correctly rejected by the Rust tool.

• Faster execution. As a native binary, avdl avoids JVM startup overhead. On real-world .avdl files it completes in single-digit milliseconds — roughly 50× faster than avro-tools idl for typical inputs, narrowing to ~6× on unusually large (1 MB) files where JVM startup is a smaller fraction of the total.

Attribution

This is a "powercoded" port of the Java Avro tools — meaning it was built with LLM assistance, incremental human guidance and review, and strong testing against the upstream test suite.

All credit for Avro itself, the original tooling, and the underlying test suite belongs to the Apache Avro project.

Key upstream sources this port is based on:

• IdlTool.java — the Java avro-tools idl entry point
• IdlReader.java — the Java IDL transformer
• Idl.g4 — the ANTLR grammar for Avro IDL

Background

The initial version of this crate was built during a live stream on YouTube. The live stream ends at commit fac28dd. Post-stream development is documented in post-stream-summary.md.

License

Licensed under the Apache License, Version 2.0.