Model building and change tracking changes to avoid loading vector properties#37829
Model building and change tracking changes to avoid loading vector properties#37829AndriySvyryd wants to merge 2 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds core infrastructure to support partial (non-auto) property loading, allowing properties to be marked as not automatically loaded and then treated as “not loaded” by change tracking/update generation (initially aimed at vector properties).
Changes:
- Introduces
IsAutoLoadedmetadata on properties (mutable/convention/runtime) plus conventions/validation for disallowed scenarios (keys/FKs/concurrency/discriminator, JSON-mapped, Cosmos). - Adds change-tracking/update support for “not loaded” scalar properties (
PropertyEntry.IsLoaded,IUpdateEntry.IsLoaded, new internal state flag) and adjusts update SQL generation to skip unloaded properties. - Adds SQL Server convention to mark vector properties as not auto-loaded by default, plus extensive test/baseline updates.
Reviewed changes
Copilot reviewed 59 out of 62 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/EFCore.Tests/Metadata/Internal/PropertyTest.cs | Adds unit tests for Property.IsAutoLoaded default and toggling. |
| test/EFCore.Tests/Metadata/Internal/InternalPropertyBuilderTest.cs | Tests configuration-source precedence for IsAutoLoaded. |
| test/EFCore.Tests/Metadata/Conventions/ConventionDispatcherTest.cs | Verifies new OnPropertyAutoLoadChanged convention dispatch behavior. |
| test/EFCore.Tests/Infrastructure/ModelValidatorTest.cs | Adds validation tests for disallowed IsAutoLoaded = false scenarios (key/FK/etc.). |
| test/EFCore.Tests/ExceptionTest.cs | Updates fake IUpdateEntry implementation for new interface member. |
| test/EFCore.Tests/ChangeTracking/PropertyEntryTest.cs | Adds change-tracking tests around unloaded properties and sentinels. |
| test/EFCore.Tests/ChangeTracking/Internal/StateDataTest.cs | Extends internal state-flag manipulation tests for new property flag. |
| test/EFCore.Sqlite.FunctionalTests/Scaffolding/Baselines/No_NativeAOT/ManyTypesEntityType.cs | Updates compiled-model baseline to emit autoLoaded: false. |
| test/EFCore.Sqlite.FunctionalTests/Scaffolding/Baselines/BigModel_with_JSON_columns/ManyTypesEntityType.cs | Same baseline update for big model (JSON columns variant). |
| test/EFCore.Sqlite.FunctionalTests/Scaffolding/Baselines/BigModel/ManyTypesEntityType.cs | Same baseline update for big model. |
| test/EFCore.SqlServer.Tests/Metadata/Conventions/SqlServerAutoLoadConventionTest.cs | New tests for SQL Server vector autoload convention behavior. |
| test/EFCore.SqlServer.Tests/Infrastructure/SqlServerModelValidatorTest.cs | Adjusts expected validation error around JSON/vector/autoload interaction. |
| test/EFCore.SqlServer.FunctionalTests/Update/NonSharedModelUpdatesSqlServerTest.cs | Adds provider-specific SQL baselines validating unloaded columns are skipped on UPDATE. |
| test/EFCore.SqlServer.FunctionalTests/Scaffolding/Baselines/No_NativeAOT/ManyTypesEntityType.cs | Updates compiled-model baseline to emit autoLoaded: false. |
| test/EFCore.SqlServer.FunctionalTests/Scaffolding/Baselines/BigModel_with_JSON_columns/ManyTypesEntityType.cs | Same baseline update for big model (JSON columns variant). |
| test/EFCore.SqlServer.FunctionalTests/Scaffolding/Baselines/BigModel/ManyTypesEntityType.cs | Same baseline update for big model. |
| test/EFCore.Specification.Tests/Scaffolding/CompiledModelTestBase.cs | Configures a property as not auto-loaded and asserts compiled model reflects it. |
| test/EFCore.Relational.Tests/Infrastructure/RelationalModelValidatorTest.cs | Adds relational validation for JSON-mapped properties disallowing not-auto-loaded. |
| test/EFCore.Relational.Specification.Tests/Update/NonSharedModelUpdatesTestBase.cs | Adds cross-provider tests verifying UPDATE skips unloaded scalar/primitive-collection columns. |
| test/EFCore.Relational.Specification.Tests/Scaffolding/CompiledModelRelationalTestBase.cs | Asserts compiled relational model contains IsAutoLoaded = false. |
| test/EFCore.InMemory.FunctionalTests/Scaffolding/Baselines/No_NativeAOT/ManyTypesEntityType.cs | Updates compiled-model baseline to emit autoLoaded: false. |
| test/EFCore.InMemory.FunctionalTests/Scaffolding/Baselines/BigModel/ManyTypesEntityType.cs | Same baseline update for big model. |
| test/EFCore.Cosmos.Tests/Infrastructure/CosmosModelValidatorTest.cs | Adds Cosmos validation test disallowing not-auto-loaded properties. |
| test/EFCore.Cosmos.FunctionalTests/TestUtilities/CosmosTestStore.cs | Updates fake IUpdateEntry implementation for new interface member. |
| test/EFCore.Cosmos.FunctionalTests/Scaffolding/CompiledModelCosmosTest.cs | Forces Cosmos compiled model to keep property auto-loaded (provider doesn’t support feature). |
| src/EFCore/Update/IUpdateEntry.cs | Adds IsLoaded(IProperty) contract used by update pipeline. |
| src/EFCore/Properties/CoreStrings.resx | Adds new core validation messages for auto-load restrictions. |
| src/EFCore/Properties/CoreStrings.Designer.cs | Generated accessors for new core validation messages. |
| src/EFCore/Metadata/RuntimeTypeBase.cs | Extends runtime property creation API with autoLoaded argument. |
| src/EFCore/Metadata/RuntimeProperty.cs | Stores runtime IsAutoLoaded and exposes it via IReadOnlyProperty. |
| src/EFCore/Metadata/Internal/Property.cs | Adds mutable/convention IsAutoLoaded metadata with configuration source tracking + convention dispatch. |
| src/EFCore/Metadata/Internal/InternalPropertyBuilder.cs | Adds builder APIs to configure IsAutoLoaded and copy it when rebuilding properties. |
| src/EFCore/Metadata/IReadOnlyProperty.cs | Introduces IsAutoLoaded (default true) and includes it in debug string output. |
| src/EFCore/Metadata/IMutableProperty.cs | Adds mutable IsAutoLoaded surface. |
| src/EFCore/Metadata/IConventionProperty.cs | Adds convention SetIsAutoLoaded and configuration source accessor. |
| src/EFCore/Metadata/Conventions/RuntimeModelConvention.cs | Ensures runtime model captures IsAutoLoaded into RuntimeProperty. |
| src/EFCore/Metadata/Conventions/Internal/ConventionDispatcher.cs | Adds dispatch point for property auto-load changes. |
| src/EFCore/Metadata/Conventions/Internal/ConventionDispatcher.ImmediateConventionScope.cs | Implements immediate execution for property auto-load changed conventions. |
| src/EFCore/Metadata/Conventions/Internal/ConventionDispatcher.DelayedConventionScope.cs | Adds delayed-scope node for property auto-load change. |
| src/EFCore/Metadata/Conventions/Internal/ConventionDispatcher.ConventionScope.cs | Adds abstract hook for property auto-load change. |
| src/EFCore/Metadata/Conventions/Infrastructure/ProviderConventionSetBuilder.cs | Registers new AutoLoadConvention in the default convention set. |
| src/EFCore/Metadata/Conventions/IPropertyAutoLoadChangedConvention.cs | New convention interface for reacting to auto-load changes. |
| src/EFCore/Metadata/Conventions/ConventionSet.cs | Wires up storage/replace/remove for IPropertyAutoLoadChangedConvention. |
| src/EFCore/Metadata/Conventions/AutoLoadConvention.cs | New model-finalizing convention that can set IsAutoLoaded based on provider heuristics. |
| src/EFCore/Metadata/Builders/IConventionPropertyBuilder.cs | Adds IsAutoLoaded fluent configuration for conventions. |
| src/EFCore/Infrastructure/ModelValidator.cs | Adds base validation blocking not-auto-loaded for key/FK/concurrency/discriminator. |
| src/EFCore/ChangeTracking/PropertyEntry.cs | Adds PropertyEntry.IsLoaded API for scalar properties. |
| src/EFCore/ChangeTracking/Internal/InternalEntryBase.cs | Tracks per-property “not loaded” state and integrates it with modified-state logic. |
| src/EFCore/ChangeTracking/Internal/InternalEntryBase.StateData.cs | Adds new IsPropertyNotLoaded state flag. |
| src/EFCore/ChangeTracking/Internal/IInternalEntry.cs | Extends internal entry contract with scalar-property IsLoaded/SetIsLoaded. |
| src/EFCore/ChangeTracking/Internal/ChangeDetector.cs | Adjusts change detection to skip unloaded properties and detect transition away from sentinel. |
| src/EFCore.SqlServer/Metadata/Conventions/SqlServerConventionSetBuilder.cs | Replaces base autoload convention with SQL Server-specific version. |
| src/EFCore.SqlServer/Metadata/Conventions/SqlServerAutoLoadConvention.cs | New convention marking vector properties as not auto-loaded by default. |
| src/EFCore.SqlServer/Infrastructure/Internal/SqlServerModelValidator.cs | Reorders vector validation relative to base property validation. |
| src/EFCore.Relational/Update/ModificationCommand.cs | Skips writing column values for unloaded properties. |
| src/EFCore.Relational/Properties/RelationalStrings.resx | Adds relational validation message for JSON-mapped properties and autoload. |
| src/EFCore.Relational/Properties/RelationalStrings.Designer.cs | Generated accessor for new relational validation message. |
| src/EFCore.Relational/Infrastructure/RelationalModelValidator.cs | Adds relational validation blocking not-auto-loaded properties for JSON mapping. |
| src/EFCore.Design/Scaffolding/Internal/CSharpRuntimeModelCodeGenerator.cs | Emits autoLoaded: false into generated compiled models. |
| src/EFCore.Cosmos/Properties/CosmosStrings.resx | Adds Cosmos validation message disallowing partial property loading. |
| src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs | Generated accessor for new Cosmos validation message. |
| src/EFCore.Cosmos/Infrastructure/Internal/CosmosModelValidator.cs | Adds Cosmos validation blocking any not-auto-loaded properties. |
Files not reviewed (3)
- src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported
- src/EFCore.Relational/Properties/RelationalStrings.Designer.cs: Language not supported
- src/EFCore/Properties/CoreStrings.Designer.cs: Language not supported
test/EFCore.Specification.Tests/Scaffolding/CompiledModelTestBase.cs
Outdated
Show resolved
Hide resolved
test/EFCore.SqlServer.Tests/Infrastructure/SqlServerModelValidatorTest.cs
Show resolved
Hide resolved
Adds infrastructure for **partial property loading** — the ability to mark entity properties as "not auto-loaded" so they can be excluded from queries and skipped during change tracking. This is the foundation for lazy-loading individual properties (e.g., large BLOBs, vectors) without loading the entire entity. After this is checked in query should provide sentinel values for not loaded properties. Cosmos support not implemented. Vector-specific model building API will be added as part of #36350 Model building Fluent API, explicit loading API and query overrides (Include) will be added in #1387
roji
left a comment
There was a problem hiding this comment.
Great to see ths - see some minor comments. Will work on the query side once this is merged.
| /// any release. You should only use it directly in your code with extreme caution and knowing that | ||
| /// doing so can result in application failures when updating to a new Entity Framework Core release. | ||
| /// </remarks> | ||
| IsPropertyNotLoaded = 6 |
There was a problem hiding this comment.
Consider inverting the logic here, i.e. having IsPropertyLoaded which defaults to true - IsPropertyNotLoaded means double negation by default (loaded) which is confusing.
There was a problem hiding this comment.
It's more efficient as most properties are loaded, so we don't need to change from the default value. Also, this distinguishes it from IsLoaded above. In any case, this is something that noone will ever look at again until we add another flag (unlikely)
| /// <remarks> | ||
| /// See <see href="https://aka.ms/efcore-docs-conventions">Model building conventions</see> for more information and examples. | ||
| /// </remarks> | ||
| public class AutoLoadConvention : IModelFinalizingConvention |
There was a problem hiding this comment.
Not sure what our regular practice here is, but is it worth having a convention that does nothing by default, just as a base class? It seems like this doesn't add much value beyond just having each provider add its own convention...
Also, don't you generally prefer avoiding IModelFinalizingConvention and doing this kind of thing reactively?
There was a problem hiding this comment.
Not sure what our regular practice here is, but is it worth having a convention that does nothing by default, just as a base class? It seems like this doesn't add much value beyond just having each provider add its own convention...
Provider-specific conventions won't be needed after #36350. Adding it here now prevents a provider-break later.
Also, don't you generally prefer avoiding IModelFinalizingConvention and doing this kind of thing reactively?
Yes, but again, that requires vector-specific API (#36350)
| ttl1, entityType1, entityType2, ttl2, container); | ||
|
|
||
| /// <summary> | ||
| /// The property '{property}' on type '{type}' cannot be configured as not auto-loaded. The Cosmos provider stores entire documents as JSON, so partial property loading is not supported. |
There was a problem hiding this comment.
I posted a question on the Cosmos channel to ask if there's anything we can do here. But in any case, I'm guessing the only practical guidance to users would be to split their embedding to a separate item (possibly in a separate container), which you only use when doing similarity search; this way, all other workloads are oblivious of embeddings, and when you do similarity search you do one search and then one ReadItem point read.
Maybe add a sentence about this to the user (ideally Cosmos would simply have this guidance in their docs and we'd point to it)
There was a problem hiding this comment.
I mainly just scoped this out for 11. We can mention this in the documentation
src/EFCore.SqlServer/Metadata/Conventions/SqlServerAutoLoadConvention.cs
Outdated
Show resolved
Hide resolved
|
|
||
| // Fall back to CLR type check when type mapping hasn't been resolved yet | ||
| return property.GetValueConverter() == null | ||
| && property.ClrType.TryGetElementType(typeof(SqlVector<>)) is null; |
There was a problem hiding this comment.
I never fully understood why type mappings are sometimes unavailable in conventions, but matching on the type like this isn't great - we have an issue for allowing more types here (#37025), and it would be better not to have to update this convention... Plus in theory someone could value-convert to a vector type (very contrived, I admit).
test/EFCore.Cosmos.FunctionalTests/Scaffolding/CompiledModelCosmosTest.cs
Outdated
Show resolved
Hide resolved
| onModelCreating: mb => | ||
| { | ||
| mb.Entity<BlogWithDescription>( | ||
| b => |
There was a problem hiding this comment.
nit: remove curlies and put on one line (elsewhere too)
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 58 out of 61 changed files in this pull request and generated 5 comments.
Files not reviewed (3)
- src/EFCore.Cosmos/Properties/CosmosStrings.Designer.cs: Language not supported
- src/EFCore.Relational/Properties/RelationalStrings.Designer.cs: Language not supported
- src/EFCore/Properties/CoreStrings.Designer.cs: Language not supported
| /// </remarks> | ||
| /// <remarks> |
There was a problem hiding this comment.
The XML doc comment has two consecutive sections (lines 11-16). Duplicate XML doc tags can produce documentation warnings/errors; merge these into a single block (and keep the constructor sentence inside it, or move it to
/ as appropriate).
Suggested change
/// </remarks>
/// <remarks>
| /// </remarks> | |
| /// <remarks> |
| /// <remarks> | ||
| /// Creates a new instance of <see cref="SqlServerAutoLoadConvention" />. | ||
| /// </remarks> |
There was a problem hiding this comment.
The XML doc comment contains duplicate tags (lines 13-18). This can trigger XML documentation warnings; combine into a single element (or move the constructor note elsewhere).
| /// <remarks> | |
| /// Creates a new instance of <see cref="SqlServerAutoLoadConvention" />. | |
| /// </remarks> |
| public virtual bool? SetIsAutoLoaded(bool? autoLoaded, ConfigurationSource configurationSource) | ||
| { | ||
| EnsureMutable(); | ||
|
|
||
| var isChanging = IsAutoLoaded != autoLoaded; | ||
| if (isChanging) | ||
| { | ||
| _isAutoLoaded = autoLoaded; | ||
| } | ||
|
|
||
| _isAutoLoadedConfigurationSource = autoLoaded == null | ||
| ? null | ||
| : configurationSource.Max(_isAutoLoadedConfigurationSource); | ||
|
|
||
| return isChanging | ||
| ? DeclaringType.Model.ConventionDispatcher.OnPropertyAutoLoadChanged(Builder) | ||
| : autoLoaded; |
There was a problem hiding this comment.
SetIsAutoLoaded uses IsAutoLoaded != autoLoaded to detect changes. When autoLoaded is null (reset to default), this comparison is always true (bool vs bool?), so conventions will run even if _isAutoLoaded is already null and no effective change occurred. Consider comparing _isAutoLoaded to autoLoaded (both nullable) or otherwise special-casing null resets so OnPropertyAutoLoadChanged only runs when the stored value actually changes.
| /// <summary> | ||
| /// The property '{property}' on type '{type}' cannot be configured as not auto-loaded. The Cosmos provider stores entire documents as JSON, so partial property loading is not supported. | ||
| /// </summary> | ||
| public static string AutoLoadedCosmosProperty(object? property, object? type) | ||
| => string.Format( | ||
| GetString("AutoLoadedCosmosProperty", nameof(property), nameof(type)), | ||
| property, type); |
There was a problem hiding this comment.
The XML doc summary for AutoLoadedCosmosProperty doesn't match the actual resource string in CosmosStrings.resx (the summary mentions storing JSON documents, while the resx value doesn't). Since this file is auto-generated, regenerate CosmosStrings.Designer.cs from the .resx (or update the .resx text to match) to keep them consistent.
| return false; | ||
| } | ||
|
|
||
There was a problem hiding this comment.
Line 210 contains trailing whitespace. Please remove it to keep formatting clean and avoid noisy diffs in future edits.
Fixes #37278
Part of #37277
Adds infrastructure for partial property loading — the ability to mark entity properties as "not auto-loaded" so they can be excluded from queries and skipped during change tracking. This is the foundation for lazy-loading individual properties (e.g., large BLOBs, vectors) without loading the entire entity.
After this Query should provide sentinel values for not loaded properties.
Cosmos support not implemented.
Vector-specific model building API will be added as part of #36350
Model building Fluent API, explicit loading API and query overrides (Include) will be added in #1387