Add APIs to `BlobBuilder` for customizing the underlying byte array et al. #115294

teo-tsirpanis · 2025-05-05T02:04:11Z

Note

Depends on #111292

This PR builds on top of @jaredpar's branch to add APIs for customizing the underlying buffer of a BlobBuilder. The chunking logic of BlobBuilder was updated to allocate multiple additional chunks with a user-customizable maximum size each. As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

This commit introduces the `System.Text.Unicode.Utf8` type to the `Microsoft.Bcl.Memory` library. It includes type forwarding for `Utf8` in `Microsoft.Bcl.Memory.Forwards.cs`, updates the documentation in `PACKAGE.md` to include `Utf8` functionality, and adds corresponding test cases in `Microsoft.Bcl.Memory.Tests.csproj`. The documentation now emphasizes `Utf8` alongside `Index`, `Range`, and `Base64Url`, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

Updated `PackageDescription` to include "Utf8" support. Added new `ItemGroup` for conditional compilation of UTF-8 and Unicode handling files for non-net8.0 frameworks. Modified visibility and implementations in `Ascii.Utility.Helpers.cs`, `Utf8.cs`, and `Utf8Utility` based on `MICROSOFT_BCL_MEMORY` define.

Updated `Microsoft.Bcl.Memory.Tests.csproj` to include `UnicodeUtility.cs` and removed .NET 8.0 targeting condition. Modified `Utf8Tests.cs` by adjusting using directives and enhancing the `DecodeHex` method with conditional compilation for .NET 5.0+.

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

Co-authored-by: Theodore Tsirpanis <[email protected]>

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

(cherry picked from commit bf6f989)

(cherry picked from commit 445a232)

Co-authored-by: Jared Parsons <[email protected]>

…iteUTF8`.

Fixes dotnet#99244.

And add more tests.

dotnet-issue-labeler · 2025-05-05T02:04:19Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

dotnet-issue-labeler · 2025-05-05T02:04:21Z

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

Copilot

Pull Request Overview

This PR adds new APIs to BlobBuilder for customizing the underlying byte array and updates related encoding and buffer-handling logic across metadata and core libraries. Key changes include replacing legacy UTF-8 encoding code with calls to the new System.Text.Unicode.Utf8 APIs, updating BlobBuilder’s API surface (including new constructors and properties), and adding NET-specific intrinsics support across several core modules.

Reviewed Changes

Copilot reviewed 31 out of 36 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
System/Reflection/Internal/Utilities/StreamExtensions.cs	Removed obsolete TryReadAll overload for Span to rely on newer API paths.
System/Reflection/Internal/Utilities/BlobUtilities.cs	Rewrote WriteUtf8 to use Utf8.FromUtf16 for encoding, replacing manual UTF-8 encoding logic.
System/Reflection/Metadata.cs	Added new BlobBuilder constructors, properties, and APIs including ReadOnlySpan/WriteBytes overloads.
System.Private.CoreLib (various files)	Updated intrinsics and preprocessor conditions (#if NET, #if SYSTEM_PRIVATE_CORELIB) for newer vectorized and ASCII helper routines.
Microsoft.Bcl.Memory (PACKAGE.md and others)	Updated documentation and type forwarding to include UTF-8 APIs for NET platforms.

Files not reviewed (5)

src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj: Language not supported
src/libraries/Microsoft.Bcl.Memory/tests/Microsoft.Bcl.Memory.Tests.csproj: Language not supported
src/libraries/System.Reflection.Metadata/System.Reflection.Metadata.sln: Language not supported
src/libraries/System.Reflection.Metadata/src/Resources/Strings.resx: Language not supported
src/libraries/System.Reflection.Metadata/src/System.Reflection.Metadata.csproj: Language not supported

...braries/System.Reflection.Metadata/src/System/Reflection/Internal/Utilities/BlobUtilities.cs

Copilot · 2025-05-05T02:04:57Z

src/libraries/System.Reflection.Metadata/ref/System.Reflection.Metadata.cs

@@ -253,6 +258,7 @@ public void WriteBytes(byte[] buffer) { }
        public void WriteBytes(byte[] buffer, int start, int byteCount) { }
        public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer) { }
        public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer, int start, int byteCount) { }


[nitpick] The newly added WriteBytes(ReadOnlySpan) API lacks XML documentation. Adding a summary and usage notes would help API consumers understand its behavior and intended usage.

Suggested change

public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer, int start, int byteCount) { }

public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer, int start, int byteCount) { }

/// <summary>

/// Writes the contents of the specified <see cref="System.ReadOnlySpan{T}"/> of bytes to the current instance.

/// </summary>

/// <param name="buffer">The span of bytes to write.</param>

/// <remarks>

/// This method does not perform any validation on the contents of the buffer.

/// Ensure that the buffer contains valid data before calling this method.

/// </remarks>

dotnet-policy-service · 2025-05-05T02:04:58Z

Tagging subscribers to this area: @dotnet/area-system-reflection-metadata
See info in area-owners.md if you want to be subscribed.

jkotas · 2025-05-05T06:59:39Z

As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

Is this needed to introduce the new APIs?

System.Text.Unicode.Utf8 change introduces a new dependency for System.Reflection.Metadata on .NET Framework that will be an extra work to push through the system. It would be better to avoid bundling the two changes together in a single PR.

am11 · 2025-05-05T12:13:04Z

As part of this, we use APIs from System.Text.Unicode.Utf8 to encode UTF-8 strings, which increases performance and safety, and reduces duplicate code.

Is this needed to introduce the new APIs?

The other PR is approved and ready to merge #111292. After the merge and rebase this branch against main, those commits will disappear.

jkotas · 2025-05-05T14:54:50Z

The other PR is approved and ready to merge #111292. After the merge and rebase this branch against main, those commits will disappear.

The change that introduces System.Reflection.Metadata dependency on Microsoft.Bcl.Memory won't disappear.

AlexRadch and others added 28 commits January 11, 2025 06:11

Add compilation constant for Microsoft BCL Memory

22be776

Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.

Apply suggestions from code review

79214bd

Co-authored-by: Theodore Tsirpanis <[email protected]>

Update Microsoft.Bcl.Memory for framework compatibility

def40c4

- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.

Space

a1caf07

Remove MICROSOFT_BCL_MEMORY from project constants

695e362

Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.

Refactor Utf8Tests for .NET compatibility improvements

063359a

Use Rune.DecodeFromUtf8 on all frameworks.

b5992c6

For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)

Fix compile errors.

90a3613

(cherry picked from commit bf6f989)

Address PR feedback.

412071a

(cherry picked from commit 445a232)

Merge branch 'main' into BclMemoryUtf8

67cb05e

Remove dead code.

d986c2e

Add extensibility points for custom blob builders.

195ac88

Add APIs to BlobBuilder for customizing the underlying byte array.

b773cc5

Co-authored-by: Jared Parsons <[email protected]>

Add tests.

571a309

Support chunking in WriteBytes(byte, int) and when writing streams.

bf4694f

Use framework methods where available.

e9b5250

Use framework methods to encode to UTF-8, and support chunking in `Wr…

17860c0

…iteUTF8`.

Zero-initialize buffers returned by BlobBuilder.ReserveBytes.

d05098c

Fixes dotnet#99244.

Remove unnecessary pinning.

d629f74

Simplify chunking logic with an IBufferWriter-like pattern.

d2995ce

Fix wrong check order.

f2ffb03

And add more tests.

Use a reasonable default max chunk size.

ca1ab97

Double the size of each new chunk until we get to the max chunk size.

e17afd4

Merge branch 'BclMemoryUtf8' into srm-buffers

eb1b0e0

Check that buffer size is not too small.

0638427

Copilot AI review requested due to automatic review settings May 5, 2025 02:04

dotnet-issue-labeler bot added the area-System.Reflection.Metadata label May 5, 2025

dotnet-issue-labeler bot added the new-api-needs-documentation label May 5, 2025

teo-tsirpanis marked this pull request as draft May 5, 2025 02:04

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label May 5, 2025

Copilot AI reviewed May 5, 2025

View reviewed changes

steveharter self-assigned this May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add APIs to `BlobBuilder` for customizing the underlying byte array et al. #115294

Add APIs to `BlobBuilder` for customizing the underlying byte array et al. #115294

teo-tsirpanis commented May 5, 2025

dotnet-issue-labeler bot commented May 5, 2025

dotnet-issue-labeler bot commented May 5, 2025

Copilot AI left a comment

Copilot AI May 5, 2025

dotnet-policy-service bot commented May 5, 2025

jkotas commented May 5, 2025

am11 commented May 5, 2025

jkotas commented May 5, 2025

-        public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer, int start, int byteCount) { }
+        public void WriteBytes(System.Collections.Immutable.ImmutableArray<byte> buffer, int start, int byteCount) { }
+        /// <summary>
+        /// Writes the contents of the specified <see cref="System.ReadOnlySpan{T}"/> of bytes to the current instance.
+        /// </summary>
+        /// <param name="buffer">The span of bytes to write.</param>
+        /// <remarks>
+        /// This method does not perform any validation on the contents of the buffer.
+        /// Ensure that the buffer contains valid data before calling this method.
+        /// </remarks>

Add APIs to BlobBuilder for customizing the underlying byte array et al. #115294

Are you sure you want to change the base?

Add APIs to BlobBuilder for customizing the underlying byte array et al. #115294

Conversation

teo-tsirpanis commented May 5, 2025

dotnet-issue-labeler bot commented May 5, 2025

dotnet-issue-labeler bot commented May 5, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI May 5, 2025

Choose a reason for hiding this comment

dotnet-policy-service bot commented May 5, 2025

jkotas commented May 5, 2025

am11 commented May 5, 2025

jkotas commented May 5, 2025

Add APIs to `BlobBuilder` for customizing the underlying byte array et al. #115294

Add APIs to `BlobBuilder` for customizing the underlying byte array et al. #115294