Random scratch ideas for how HybridCache could implement the someFormatterThing efficiently, using interceptors to replace a formattable string with a custom formatter: https://gist.github.com/mgravell/69ec4c1a23fbaf7bce0277f86d8ad1d4 by turning $"this {key.id:000} is {key.region} magic" into something that emits:

var scratch = new Writer<TWriter>(ref writer);
scratch.WriteString("this ");
scratch.WriteSpanFormattable(key.id, "000", provider);
scratch.WriteString(" is ");
scratch.WriteString(key.region);
scratch.WriteString(" magic");
scratch.Flush();

(but everything still works if that magic isn't available)

Random scratch ideas for how HybridCache could implement the someFormatterThing efficiently

Why not use an interpolated string handler?

@stephentoub because I didn't know that was a thing; if that is a good fit here, great! Huge thanks! Linking for the long tail (and to check we're on the same page): https://learn.microsoft.com/dotnet/csharp/advanced-topics/performance/interpolated-string-handler

Yup. I only briefly skimmed what you wrote, but on quick perusal it looks like a good fit.
https://devblogs.microsoft.com/dotnet/string-interpolation-in-c-10-and-net-6/

@stephentoub yes, looks like an ideal fit; in fact, wait one moment for a follow-up API suggestion... because right now I can get everything I want via:

var key = (id: 42, region: "abc");
var cache = new Cache();
cache.Get<int>($"abc {key.id:000} in {key.region}");

...

class Cache
{
    public T Get<T>(ref DefaultInterpolatedStringHandler key) // real code has "miss" callback etc
    {
        // this is just to show it is working; obvs we don't actually ToString usually
        Console.WriteLine($"Fetching: {GetText(ref handler).ToString()}");
        Clear(ref handler);

        return default!; // result doesn't matter here
    }

    [UnsafeAccessor(UnsafeAccessorKind.Method, Name = "Clear")]
    private static extern void Clear(ref DefaultInterpolatedStringHandler value);

    [UnsafeAccessor(UnsafeAccessorKind.Method, Name = "get_Text")]
    private static extern ReadOnlySpan<char> GetText(ref DefaultInterpolatedStringHandler value);
}

Tagging subscribers to this area: @dotnet/area-extensions-caching
See info in area-owners.md if you want to be subscribed.

// this is just to show it is working; obvs we don't actually ToString usually

ReadOnlySpan<char> should work fine as an interpolated string argument on core... did it not?

in fact, wait one moment for a follow-up API suggestion... because right now I can get everything I want via:

MemoryCache builds for netstandard and netfx as well, and DefaultInterpolatedStringHandler isn't available there. Are you proposing these new APIs only for .NET Core builds?

I am trying to envision what the tuple/record/etc. case look like to avoid the case of the handler as well we initially were thinking here - does this approach mean we would need to figure this our per cache type, or do we want to think through this in a more holistic way given the underlying split of string/everything-else split that's happened beneath in several places now?

I guess the other argument to be made for usability is records being a good fit - perhaps it's worth seeing what is the ideal key scenario across all these cache types with usability vs. hashing and imagine what that might look like equally on say MemoryCache and HybridCache as well?

I have been thinking of this in terms of how we can access caches directly today with more efficient lookups, but not most of the caching APIs, for example:

void Main()
{
    Util.AutoScrollResults = true;
    BenchmarkRunner.Run<LookupBenchmarks>();
}

[MemoryDiagnoser]
public class LookupBenchmarks
{
    private ConcurrentDictionary<(UserType UT, RequestType RT, string RequestID), object> _tupleCache = new();
    private ConcurrentDictionary<RequestKey, object> _recordCache = new();

    [Benchmark]
    public object TupleLookup() => _tupleCache.TryGetValue((UserType.A, RequestType.B, "myKey"), out var val) ? val : default;
    [Benchmark]
    public object RecordLookup() => _recordCache.TryGetValue(new(UserType.A, RequestType.B, "myKey"), out var val) ? val : default;
}

public enum UserType { A, B }
public enum RequestType { A, B, C }
public readonly record struct RequestKey(UserType UT, RequestType RT, string RequestID);

...but I realize everything today is a string on the common caching APIs, so maybe that's the only viable path without API explosion. If they all could take things like a readonly record struct though...possibilities.

@NickCraver The interpolated string handler suggested replaces the need to even worry about that - and has the advantage of working with the L1 we have today (if we make the change proposed here) rather than some speculative L1 we might wish to have in the future (RCache etc with TKey generic key usage). So your usage just becomes regular HybridCache:

HybridCache cache = ... // Probably injected

var data = cache.GetOrCreateAsync($"/foos/{region}/{id}/bars", ...callback...);

That work? Note this would be based on the alt-lookup if CD<string,object> which means: .NET 9+ if I hack in unsafe-accessor, and .NET 10+ to use the new API if we get it.

What is the comparable performance like for that case? And is the proposed API overload in that case only an interpolated string arg? (I'm not sure how that would be the case, I imagine this has to be a priority underload?)

My worry here is that the existing APIs are strings, so what prevents a user from doing the wrong thing, constructing the string earlier, passing it in method calls, etc. and not actually getting the interpolated string overload by the time they get here? One goal, I think, should be the inability to hold it wrong, and I'm not seeing how we'd accomplish that here - thoughts?

Great question. For same-method usage we could potentially add an analyzer that spots string locals populated in the same method. In the wider usage, at some point we have say that if someone cares enough to want to remove string allocs, they're going to have to check that they're using it correctly. And yes: interpolated string handlers are priority overloads when passed an interpolated string literal.

Gotcha. So overall: I would likely not use this interpolated-based API if available, as it just as easily leads devs down a lot of the same bad roads we have today. A big win, in my mind with the above options is the stronger typing of the key and obviousness of the code.

For example, if someone does $"{"aa"}{1}{22}", it's the same as $"{"a"}{12}{2}". Of course, that's no worse than today forming the strings, but it's no better either. I just want us to consider all the pitfalls this has that we find in production code with formatting, locales, collisions, etc.

I think we have a larger opportunity here to make cache keys be effectively typed, in ways that eliminate these types of errors while being more efficient as well. But to do that, we have to not be tied to all the old string overloads as the only funnel. If we're constrained to that, of course I don't object to optimizing it some, but it seems like we're leaving a lot of the pit-of-success (in both correctness and performance) on the table, and I'd be disappointed if we left it there.

If we zoom out, this non-interpolated path also works for things like collections of keys from cache, which matters more so in remote fetch scenarios where we're batching. Maybe there's a clean way to do batch APIs with interpolated, but I'm having trouble picturing it. And admittedly we can solve some of what I'm talking about for instance by banning the string-based overload as an API, but I'm not sure that's a friendly or practical solution to force optimal behavior because we just as often have at least one "GlobalCachedThing" as a non-parameter key.

Your points there re key validity do actually make it very tempting to wrap the default impl in a ref struct KeyBuilder or similar, allowing me to enforce custom rules at either runtime or build-time (analyzer) - things like "formatted arguments must be separated by literals". I'd still need to get at the internal impl, though, so it doesn't change the need for the API

@mgravell is this something that is blocking HybridCache v1 ATM or is it a nice-to-have, that could be implemented after its release? I'm assuming the latter but was curious.

Definitely the latter; the initial release will be string key, but in vFuture it would be nice if we can create a zero-alloc path for the hot "local cache hit" path using composed keys. To do that, we need to be able to ask the IMemoryCache (I'm fine with type-testing against the concrete MemoryCache) for values using a ReadOnlySpan<char>.

Here's a fully worked example of how two proposed tweaks combine to allow complex cache scenarios to use alloc-free code paths, including retroactively when a string API already exists: https://gist.github.com/mgravell/750bed134f36f417f97c217297552a88

@mgravell I got the impression that if this is approved, you would implement it. With that assumption, I've put it into the .NET 10 milestone.

Video

• There was an alternative proposal involving interpolated string handlers. It is mothballed for now.
• OverloadResolutionPriority is required since strings are now equally ReadOnlySpan<char> and object.
• It was asked if GetOrCreate should also be overloaded at this time, and the answer was no (for now).
• The new methods will not be available in the .NET Standard TFM, due to a lack of IAlternateLookup.
  • .NET 9+

namespace Microsoft.Extensions.Caching.Memory
{
  public class MemoryCache : IMemoryCache
  {
      [OverloadResolutionPriority(1)]
      public bool TryGetValue(ReadOnlySpan<char> key, out object? value);
      [OverloadResolutionPriority(1)]
      public bool TryGetValue<TItem>(ReadOnlySpan<char> key, out TItem? value);
  }
}