Add Semantic Similarity chunker #6994

KrystofS · 2025-10-31T19:25:13Z

Microsoft Reviewers: Open in CodeFlow

KrystofS · 2025-10-31T19:27:44Z

@KrystofS please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@dotnet-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@dotnet-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@dotnet-policy-service agree company="Microsoft"
Contributor License Agreement

@dotnet-policy-service agree

Copilot

Pull Request Overview

This PR introduces new chunking implementations for the DataIngestion library, adding support for splitting documents into chunks based on different strategies: semantic similarity, sections, markdown structure, and token-based chunking. The key changes include:

New chunker implementations: SemanticSimilarityChunker, SectionChunker, MarkdownChunker, and DocumentTokenChunker
Comprehensive test coverage for all new chunkers
Addition of System.Numerics.Tensors package dependency for cosine similarity calculations
Addition of Microsoft.ML.Tokenizers.Data.O200kBase package for tokenization support
Test infrastructure with shared TestEmbeddingGenerator for testing AI-based chunking

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`src/Libraries/Microsoft.Extensions.DataIngestion/Microsoft.Extensions.DataIngestion.csproj`	Adds `System.Numerics.Tensors` package dependency and removes trailing whitespace
`src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SemanticSimilarityChunker.cs`	Implements semantic similarity-based chunking using embeddings and cosine similarity
`src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs`	Implements section-based chunking that treats document sections as separate entities
`src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/MarkdownChunker.cs`	Implements markdown header-based chunking with configurable header levels
`src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/DocumentTokenChunker.cs`	Implements token-based chunking with configurable overlap
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj`	Adds test dependencies and links to shared test helpers
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SemanticSimilarityChunkerTests.cs`	Comprehensive tests for semantic similarity chunker
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SectionChunkerTests.cs`	Tests for section-based chunking including nested sections
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/MarkdownChunkerTests.cs`	Tests for markdown chunker with various header configurations
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/DocumentTokenChunkerTests.cs`	Base tests for token-based chunking
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/OverlapDocumentTokenChunkerTests.cs`	Tests for token chunking with overlap
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/NoOverlapDocumentTokenChunkerTests.cs`	Tests for token chunking without overlap
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/DocumentChunkerTests.cs`	Base test class with common test scenarios
`test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/ChunkAssertions.cs`	Helper class for chunk assertions

...ibraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SemanticSimilarityChunkerTests.cs

test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/MarkdownChunkerTests.cs

src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/MarkdownChunker.cs

adamsitnik

Big thanks for all your hard work and contribution to our product @KrystofS !

I've left some comments, PTAL.

src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/DocumentTokenChunker.cs

src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SectionChunker.cs

...raries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/OverlapDocumentTokenChunkerTests.cs

...ies/Microsoft.Extensions.DataIngestion.Tests/Microsoft.Extensions.DataIngestion.Tests.csproj

...ibraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SemanticSimilarityChunkerTests.cs

adamsitnik

Overall looks good to me. PTAL at my comments (mostly nits). Thank you again @KrystofS !

src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SemanticSimilarityChunker.cs

...ibraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/SemanticSimilarityChunkerTests.cs

test/Libraries/Microsoft.Extensions.DataIngestion.Tests/Chunkers/ChunkAssertions.cs

adamsitnik

LGTM, thank you for your contribution @KrystofS !

src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/SemanticSimilarityChunker.cs

Co-authored-by: Copilot <[email protected]>

…nticSimilarityChunker.cs Co-authored-by: Adam Sitnik <[email protected]>

Co-authored-by: Adam Sitnik <[email protected]>

Copilot AI review requested due to automatic review settings October 31, 2025 19:25

github-actions bot added the area-telemetry label Oct 31, 2025

dotnet-policy-service bot assigned KrystofS Oct 31, 2025

Copilot AI reviewed Oct 31, 2025

View reviewed changes

adamsitnik reviewed Nov 3, 2025

View reviewed changes

adamsitnik reviewed Nov 5, 2025

View reviewed changes

adamsitnik approved these changes Nov 5, 2025

View reviewed changes

adamsitnik changed the title ~~Add new document chunkers~~ Add Semantic Similarity chunker Nov 5, 2025

adamsitnik requested a review from stephentoub November 5, 2025 16:16

stephentoub reviewed Nov 5, 2025

View reviewed changes

KrystofS and others added 17 commits November 5, 2025 18:04

Chunkers

b1c27a2

add tests

761843f

fix chunker tests

df3e02c

Apply suggestions from code review

d9e05f2

Co-authored-by: Copilot <[email protected]>

resolve suggestions from copilot

74177de

Update src/Libraries/Microsoft.Extensions.DataIngestion/Chunkers/Sema…

3793004

…nticSimilarityChunker.cs Co-authored-by: Adam Sitnik <[email protected]>

Apply suggestions from code review

7d7ee81

Co-authored-by: Adam Sitnik <[email protected]>

refactors based on review

f60bf3b

fix tests

d214a95

fix missed warning

476866e

update semanctic similarity chunker documentation

d767031

fix typo

91e26a6

Update SemanticSimilarityChunker.cs

7107417

remove chunkers from PR

ada19ef

remove left over tests

d1d4028

Delete ChunkAssertions.cs

d2d736a

Apply suggestions from code review

78afabb

Co-authored-by: Adam Sitnik <[email protected]>

KrystofS force-pushed the feature/chunkers branch from 20c8bd7 to 78afabb Compare November 5, 2025 17:04

Update SemanticSimilarityChunker.cs

ae8a74a

KrystofS added 3 commits November 5, 2025 18:39

Update SemanticSimilarityChunker.cs

ca35d95

fix documentation

50f22f0

add missing period

27b9e72

adamsitnik added area-data-ingestion and removed area-telemetry labels Nov 5, 2025

adamsitnik enabled auto-merge (squash) November 5, 2025 20:19

adamsitnik merged commit 192782e into dotnet:main Nov 5, 2025
6 checks passed

Add Semantic Similarity chunker #6994

Add Semantic Similarity chunker #6994

Conversation

KrystofS commented Oct 31, 2025 • edited by dotnet-policy-service bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microsoft Reviewers: Open in CodeFlow

Uh oh!

KrystofS commented Oct 31, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamsitnik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KrystofS commented Oct 31, 2025 •

edited by dotnet-policy-service bot

Loading