I think this is all good, but I would like to better understand this:

In the above example this would mean Bar would be seen as struct { X T }

Seen by whom? I think it's important that Baz[Bar] in a context of Foo[int] are distinguished from the one in context Foo[bool], at least in the runtime (e.g. if the compiled code uses reflection to instantiate it).

Initially I intended to change the substitution to always modify the underlying struct in a named type. However, that causes problems in the instance map since the instance map uses the object pointer as a key and the fully substituted object has a different pointer.

Could you elaborate on this? It sounds like the problem is that when we are trying to compute type arguments for Baz, we incorrectly use the generic definition of Bar instead of the local instance. What if we apply type resolution to all type arguments before we attempt to instantiate a dependent nested type?

@nevkontakte Sorry, I wasn't very clear in my comment about the pointers. Instead of the example from the comment, lets use this one:

func Foo[T any]() {
	type Bar[U any] struct {
		X T
		Y U
	}
}

In the above we have the type Bar[T; U] where T is the implicit type. When we use types.Instantiate on a named type it will not perform the substitution of the underlying type (like I has mistakenly thought in the prior PR). Named types are "substituted lazily", meaning that Bar[T; int] would still have the underlying type of struct { X T; Y U }, not struct { X T; Y int } like I had thought it would. (This is what I was trying to say when I said "this would mean Bar would be seen as struct { X T }", the underlying type wouldn't be substituted like I had expected it to be.)

I had to fix the code checking for concrete types in the visitor, so that it wouldn't have false negatives caused by the underlying types. But I still wanted to detect that Bar[T; int] is generic. Since the named type only had type parameters and type arguments but no implicit/nest type parameters/arguments, we couldn't stop at a named type and assume it was concrete because it had concrete type arguments, we had to still check the underlying struct to find the T. (If the type didn't have X T, then it would be concrete because all the type parameters had been realized, even though it is nested in a generic type.)

When substituting T in Bar[T; U], a new named type with the underlying struct is created. For example, if T is substituted with bool, we would get Bar[bool; U] with the underlying type being struct { X bool; Y U }. This new named type would not have an origin type and no type arguments. It appears like a unique generic Bar[U] in every instance of Foo. Each instance would have different pointers because of the new types and underlying types.

This became an issue with how I had updated the Instance to include nesting types. Having a brand new Bar[U] object for every instance of Foo[T] caused problems because Bar[bool; U] wasn't anywhere in types.Info, only the original Bar[T; U] object is (i.e. the Bar object created by the type checker). Since we use the object in the Instances for various things (like in funcContext.knownInstances), it was easier to make the object in an Instance be that original Bar, than it would be trying to add an origin to the Instance struct and have the Instance's object point to the substituted version.

Overall this means that the whole underlying type was "substituted lazily" including the type arguments themselves. To use it, we'd have to keep a map from type parameters to type arguments, including the implicit ones from the nest, so we can substitute while processing the underlying types. Which is great since the code you had already written (i.e. the replacer) already did that; I just had to adjust how I was thinking about nested types from the prior PR to this PR and adjust the code I wrote to match these changes.

P.S. I think what tripped me up was that the substr wanted to fully substitute in most cases but in some cases with named types they used types.Instantiate that is lazy substitution, so sometimes I'd get a new pointer with everything substituted and sometimes I wouldn't. And, even if no substitution occurred in a named type, it'd create a new new copy of the named type. When making copies they had a bug in the caching that was confusing too. Most of the substr doing this (what felt like) inconsistent behavior, was when the named type was nested. The lambda calculus in the comments didn't seem to match what it actually did, but I haven't done lambda calculus since undergrad so I probably was just reading it wrong. I kept trying to use that part of the code and leverage their nesting stuff but it was complicating things. The substr also always required a function, which probably makes sense in the context of SSA but didn't for what we were doing. Or maybe I'm just not understanding what that code was doing. That's why I just eventually always passed in nil to substr for the nest function making it not use that code. Things got easier after that.

Ah, I see now, that makes sense. As an aside, in other places we haven't shied away from adding more data into types.Info, so that's also always an option.