Adds support for writing macros in IonManagedWriter_1_1 #934

popematt · 2024-09-06T17:07:07Z

Issue #, if available:

#733, #737, #742

Description of changes:

I've split it into multiple commits to make it easier to review.

Tyler's patch for IonReaderContinuableCoreBinary is a temporary thing that will end up being a merge conflict and needing to be removed once Adds support for evaluating user-defined macros from binary streams. #933 is merged.
Renames finish() to flush() in raw writer and fixes flush() implement…. This is something that was bothering me, and there was a bug in the raw text writer's implementation of finish() because it didn't actually flush the output.
Adds MacroAwareIonWriter, WriteAsIon interfaces; also adds a new method to IonWriter with a default implementation.
Small fixes to MacroRef and other macro-related things. Commit title is mostly self-explanatory. See 🗺️ notes in PR diff.
Adds MacroAddressStrategy and minor fixes for ManagedWriterOptions_1_1. Commit title is mostly self-explanatory. See 🗺️ notes in PR diff.
Implements macro support in IonManagedWriter_1_1. This is the biggest commit by far, but about 3/4ths of it is testing, so there's not quite as much logic to review as you might think.
Little bits of cleanup fixes a few things that I messed up when I was arranging my changes into logical commits for this PR.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2024-09-06T17:13:18Z

Codecov Report

Attention: Patch coverage is 84.72222% with 33 lines in your changes missing coverage. Please review.

Please upload report for BASE (ion-11-encoding@00e45de). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...va/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt	87.43%	9 Missing and 14 partials ⚠️
.../com/amazon/ion/impl/bin/IonRawBinaryWriter_1_1.kt	25.00%	3 Missing ⚠️
src/main/java/com/amazon/ion/IonWriter.java	0.00%	2 Missing ⚠️
...mazon/ion/impl/IonReaderContinuableCoreBinary.java	33.33%	1 Missing and 1 partial ⚠️
.../com/amazon/ion/impl/bin/SymbolInliningStrategy.kt	66.66%	2 Missing ⚠️
...rc/main/java/com/amazon/ion/impl/macro/MacroRef.kt	50.00%	1 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                 @@
##             ion-11-encoding     #934   +/-   ##
==================================================
  Coverage                   ?   69.87%           
  Complexity                 ?     6779           
==================================================
  Files                      ?      192           
  Lines                      ?    26808           
  Branches                   ?     4862           
==================================================
  Hits                       ?    18733           
  Misses                     ?     6574           
  Partials                   ?     1501

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

popematt · 2024-09-06T17:07:50Z

src/main/java/com/amazon/ion/IonWriter.java

    public void writeBlob(byte[] value, int start, int len)
        throws IOException;
+
+    public default void writeObject(WriteAsIon obj) {


🗺️ I'm not married to this name. Suggestions for a better name are welcome.

popematt · 2024-09-06T17:11:13Z

src/main/java/com/amazon/ion/WriteAsIon.kt

+    /**
+     * Writes this object to an IonWriter capable of producing macro invocations.
+     */
+    fun writeToMacroAware(writer: MacroAwareIonWriter) = writeTo(writer as IonWriter)


🗺️ I didn't want to overload writeTo. Naming suggestions welcome here if you can think of a better one.

popematt · 2024-09-06T17:12:14Z

src/main/java/com/amazon/ion/impl/IonRawTextWriter_1_1.kt

        if (closed) return
        confirm(depth() == 0) { "Cannot call finish() while in a container" }
        confirm(numAnnotations == 0) { "Cannot call finish with dangling annotations" }
+        output.flush()


🗺️ This was the bug in the finish() (now flush()) implementation.

popematt · 2024-09-06T17:13:56Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

 * TODO: What package does this really belong in?
- *       See also [ManagedWriterOptions], [SymbolInliningStrategy], and [DelimitedContainerStrategy].
+ *
+ * See also [ManagedWriterOptions_1_1], [SymbolInliningStrategy], and [LengthPrefixStrategy].


🗺️ Moved so that the KDoc links will work. For some reason, KDoc links didn't seem to work when the IDE sees it's in a "TODO" item.

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

popematt · 2024-09-06T17:47:35Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

+            // TODO: See if there's any benefit to using a smaller number type, if we can
+            //       memoize this in the macro definition, or replace it with a list of precomputed
+            //       step-out indices.
+            /** Tracks where and how many times to step out. */
+            val numberOfTimesToStepOut = IntArray(macro.body.size + 1)


🗺️ It just occurred to me that precomputing and/or memoizing this information in the TemplateMacro would also make it easier to break up this large method into smaller methods. The reason I didn't break this up into smaller methods is because of the awkwardness of passing around the numberOfTimesToStepOut array.

I'll leave that as a TODO for later though.

popematt · 2024-09-06T17:51:53Z

src/main/java/com/amazon/ion/impl/bin/IonRawBinaryWriter_1_1.kt

-            EEXP -> presenceBitmapStack.peek().signature[currentContainer.numChildren].type.taglessEncodingKind
+            EEXP -> {
+                val signature = presenceBitmapStack.peek().signature
+                if (currentContainer.numChildren >= signature.size) throw IllegalArgumentException("Too many arguments for macro with signature $signature")
+                signature[currentContainer.numChildren].type.taglessEncodingKind
+            }


🗺️ Previously, this would just throw an IndexOutOfBoundsException (or something like that). I ran into this unhelpful exception while I was testing things, so I put in a more useful message.

popematt · 2024-09-06T17:56:57Z

src/main/java/com/amazon/ion/impl/bin/SymbolInliningStrategy.kt

-        val NEVER_INLINE = SymbolInliningStrategy { _, _ -> false }
+        val NEVER_INLINE = object : SymbolInliningStrategy {
+            override fun shouldWriteInline(symbolKind: SymbolKind, text: String): Boolean = false
+            override fun toString(): String = "NEVER_INLINE"
+        }


🗺️ The only real difference here is that instead of a lambda expression, we have an anonymous class with a useful toString() implementation.

popematt · 2024-09-06T18:03:09Z

src/test/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1_Test.kt

+        companion object {
+            // This is a very long macro name, but by using the qualified class name,
+            // there is almost no risk of having a name conflict with another macro.
+            private val MACRO_NAME = Point2D::class.qualifiedName!!.replace(".", "_")


🗺️ It would be even safer to replace . with $, but that was not playing nicely with the expected output in the test case because Kotlin uses $ to indicate string interpolation.

popematt · 2024-09-06T18:04:32Z

src/test/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1_Test.kt

+                templateBody {
+                    struct {
+                        fieldName("x")
+                        variable(0)
+                        fieldName("y")
+                        variable(1)
+                    }
+                }


🗺️ If we want people to be able to define macros for their own types, I think we do need a builder of some sort. It's so much easier to use a builder than it is to construct the expressions directly.

popematt · 2024-09-06T18:16:17Z

src/main/java/com/amazon/ion/SystemSymbols.java

🗺️ I rearranged this file so that the Ion 1.0 symbols are first, and any added Ion 1.1 symbols come afterwards. I also added ANNOTATE, EXPORT, TDL_EXPRESSION_GROUP, LITERAL, and MAKE_SEXP.

popematt · 2024-09-06T18:16:51Z

src/test/java/com/amazon/ion/impl/macro/ExpressionBuilderDsl.kt

🗺️ There is a lot of repetitive code here. Most of it is interfaces that define what methods may be called when stepped into various container types for each of DataModelExpression, TemplateBodyExpression, and EExpressionBodyExpression. This is what lets us elegantly write things like

templateBody { list { int(1) variable(0) macro("foo") { string("bar") } } }

zslayton

LGTM! Coupla small things to consider before merging.

zslayton · 2024-09-06T18:50:07Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

+        // Check the current macro table
+        var id = macroTable[macro]
+        if (id != null) return id
+        // Check the to-be-appended macros
+        id = newMacros[macro]
+        if (id != null) return id
+        // Add to the to-be-appended symbols


Should this check the newMacros first so new definitions shadow the old ones?

No—at least not here—because this uses deep equality of the macro signature and body. (In practice, though, it should be short-circuited by a this === other check most of the time.)

However, your question is making me rethink the addMacro(...): MacroRef function because MacroRef could outlive an encoding context with no good way to determine whether it's still valid.

zslayton · 2024-09-06T18:51:15Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

+                    repeat(numberOfTimesToStepOut[index]) { stepOut() }
+                }
+
+                when (expression) {


Should the writing logic live in the WriteAsIon impl for the various expression kinds?

tgregg

Looking good!

tgregg · 2024-09-07T07:05:37Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

-            newSymbols.forEach { (text, _) -> writeString(text) }
-            stepOut()
-            stepOut()
+    private fun resetEncodingContext() {


After this is called, how does the writer know that it needs to write a new encoding directive before the next value?

The next time the user values are flushed, if newSymbols or newMacros is non-empty, then the managed writer will write an encoding directive before it flushes the user values. If both are empty, then we know that there are no SIDs or E-Expressions in the user values being flushed, and so no encoding directive is needed because it makes no difference.

Now that I think about it, there might be an edge case for Ion text if a system macro was being shadowed—it would end up writing incorrect data. This can be solved by making the managed writer emit an IVM if a value is written after finish() has been called.

tgregg · 2024-09-07T07:18:57Z

src/test/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1_Test.kt

+    }
+
+    @Test
+    fun `write an e-expression by name 2`() {


Is this just a duplicate of the previous test?

Yes, it is. Oops.

tgregg · 2024-09-07T07:36:40Z

src/main/java/com/amazon/ion/impl/bin/IonManagedWriter_1_1.kt

+                    if (parameter.cardinality != Macro.ParameterCardinality.ExactlyOne) {
+                        // TODO: Consider adding a method to the raw writer that can write a single-character
+                        //       symbol without constructing a string. It might be a minor performance improvement.
+                        // TODO: See if we can write this without a space between the parameter name and the sigil.


+1, I think this will be very important for readability.

…ation

popematt force-pushed the managed-macro-writer-pr branch 3 times, most recently from bfba2e3 to 770c81e Compare September 6, 2024 18:11

popematt commented Sep 6, 2024

View reviewed changes

popematt requested review from tgregg and zslayton September 6, 2024 18:19

zslayton approved these changes Sep 6, 2024

View reviewed changes

tgregg approved these changes Sep 7, 2024

View reviewed changes

popematt added 7 commits September 7, 2024 13:30

Renames finish() to flush() in raw writer and fixes flush() implement…

2092e8a

…ation

Adds MacroAwareIonWriter, WriteAsIon interfaces

bc37d49

Small fixes to MacroRef and other macro-related things

e4402f6

Adds MacroAddressStrategy and minor fixes for ManagedWriterOptions_1_1

400fd13

Implements macro support in IonManagedWriter_1_1

3ddb730

Little bits of cleanup

cf0b058

Additional changes based on PR feedback and rebase cleanup

a84b4ed

popematt force-pushed the managed-macro-writer-pr branch from 770c81e to a84b4ed Compare September 7, 2024 22:33

popematt merged commit c454ecc into amazon-ion:ion-11-encoding Sep 7, 2024

popematt deleted the managed-macro-writer-pr branch September 7, 2024 23:11

tgregg pushed a commit that referenced this pull request Sep 9, 2024

Adds support for writing macros in IonManagedWriter_1_1 (#934)

49769f5

popematt mentioned this pull request Sep 12, 2024

Remove MacroRef from user-facing APIs #937

Merged

tgregg pushed a commit that referenced this pull request Dec 13, 2024

Adds support for writing macros in IonManagedWriter_1_1 (#934)

9284d28

popematt added a commit to popematt/ion-java that referenced this pull request Jun 19, 2025

Adds support for writing macros in IonManagedWriter_1_1 (amazon-ion#934)

8b18afd

Adds support for writing macros in IonManagedWriter_1_1 #934

Adds support for writing macros in IonManagedWriter_1_1 #934

Uh oh!

Conversation

popematt commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zslayton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgregg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

popematt commented Sep 6, 2024 •

edited

Loading

codecov bot commented Sep 6, 2024 •

edited

Loading