I like this. But also per #8231 we'd need a way to mark a test as intentionally having an "error" type.

I sent a draft just to check how many tests contain semantic TS errors: #8351

There is really lot of them...

Test Suites: 51 failed, 129 passed, 180 total
Tests:       1643 failed, 1 skipped, 27628 passed, 29272 total

Some tests, such as dot-notation.test.ts, contain errors in almost every case..

https://github.com/typescript-eslint/typescript-eslint/pull/8351/files#diff-93cb24cbe290e1f30e97aaf0631ade9bd3751820d1f0bef8ecb4a0532243e2bd

So if we decide we want to fix all of them, that's a huge git diff (maybe we can split it into several prs)

Also, I'm not sure how best to handle cases with fixers. I suppose we'd like to be able to write both types of test cases:

1. We expect code to contain a certain ts error. But the output shouldn't contain ts errors
2. We expect code to contain a certain ts error. And the output should contain ts error

Any thoughts on this? I just want to make sure we want to fix all the failed tests before I start fixing the 1643 cases 😄

The issue is that there are a lot of tests where we don't actually care if there is a type error because the code is testing a pure syntactic code path.

For example - dot-notation only cares about types specifically in the case of private member access. So all of those tests with errors are moot - whether or not they have errors they still accurately model the 99.9% case of not being member access on a class.

So if we were to build this we would need a way to easily opt out, IMO. It's not worth us writing type-correct test cases for every single case, IMO, as that would involve multi-line tests where a single-line would have been sufficient.

1643 failed

😭.

...although, hey, 1643 / 29272 is ~5.6%. I suppose that's an 'A' grade...? 😂

a way to easily opt out, IMO

I'm now wondering whether it should be opt in or opt out... We the typescript-eslint team are very good at doing this right. But I don't know that we'd want to have that assumption hold by default for not-us folks. And even we've had slips in.

I like how #8351 has an ignoreTsErrors explicitly set. That feels good to me.

Having it key on specific error codes is a little trickier given that they aren't stable version-to-version. That'll likely cause pain down the line the next time TypeScript shuffles around error codes in a version. Which happens once in a while. And is a pity because in theory I do really like the idea of needing to specify which errors are allowed. 😞

split it into several prs

Agreed - for any big change like this I'm a favor of splitting out

given that they aren't stable version-to-version

They're a lot more stable than the TS team says they are.

You know, Jake Bailey recently asked me out of curiosity what my next big ask for the TS team is... maybe finally getting around to asking for stable error categories is up there.

I think any rule whose tested code path interacts with the TS compiler should be error-free. For dot-notation, this means any case that involves a computed access and one of the custom options. a[0]; on itself shouldn't be "valid" because the type we retrieve from TS has no semantic meaning. However, those test cases whose code path do not involve type information, I think we can opt out, though I would accept a PR that makes them all "valid". My idea is that our rule tester should have a way to register some global declarations for a single test file, such as declare a: Record<string, unknown>.

The difficult thing is that the only way to "register" types is via code. So that means injecting code into the test cases. The complicated thing is that will show up in the test runner output then which can be confusing.

That also adds the complexity of needing to ensure the test case doesn't declare the same variables that are "pre-declared"

For end-users it would probably be more convenient if this feature was opt-in. Since otherwise we may break a lot of existing tests.

I see that this feature can be enabled separately in each test case:

ruleTester.run('my-rule', rule, {
  valid: [
    {
      code: 'const x = 1;',
      checkWithTSC: true,
    },
  ],
})

Or on each RuleTester instance:

const ruleTester = new RuleTester({
  // ...
  checkTestsWithTSC: true
})

Moreover, we can add the ability to configure default behavior globally for all RuleTester instances in some test "setup" file, like for RuleTester.afterAll etc (https://typescript-eslint.io/packages/rule-tester#with-specific-frameworks)

RuleTester.checkTestsWithTSC = true

I agree that some rules don't require all tests to be error free, some of them may be sensitive to TS errors only in some specific conditions to check some rule implementation details.
But overall if we're targeting TS error-free code, I think it worth to ensure this in all our tests.

Yes, ensuring that existing test cases are free of TS errors will require a lot of work, but later on when someone adds new tests, it won't be so hard anymore. WDYT?

I think having it on by default is fine if the case uses type information.
We can wrap the parser services to detect when a case accesses type info and flick the option on for those rules.

In that case I think being breaking is fine here --- we're highlighting a case where their test case could be plain wrong. Type errors fall to any silently and thus a case can pass silently and accidentally.

But for non-type-aware cases I agree that it's not correct to be on by default. I'd even say it's plain wrong to be on at all for them. A non-type-aware rule only checks syntax so types are irrelevant and ensuring types are correct can get in the way of devx.

I agree. This proposal is solely for type-aware rules.

So, to summarize the discussion above:

If the check is not explicitly requested for a particular test case, we perform a TSC check when one of services.getSymbolAtLocation/services.getTypeAtLocation or services.program.* is accessed. Otherwise we only check if it has been explicitly requested in each individual test/each RuleTester instance.

So this would be a kind of opt-out feature, right?

About ignoring specific TS diagnostics:

The ability to ignore specific codes on a per-test basis sounds good, but as @JoshuaKGoldberg mentioned, these codes are not yet considered stable. If we look at src/compiler/diagnosticMessages.json history for 2024, 10 diagnostic messages were removed/changed, and 3 of them could potentially be listed in someone's ignore list (7 of them are related to build/project setup).
On the other hand, I think in most cases users will ignore diagnostic codes that have been around for years and are unlikely to be removed/changed.

I also don't really like the idea of using only raw diagnostic codes because it's very unintuitive. Personally, I wouldn't want to open a test file and see only ignoreTsErrors: [2362, 2363]. To figure out what does it mean, you'd have to open the diagnosticMessages.json file and find the corresponding message.

Using messages to identify a specific diagnostic (e.g. Diagnostics.The_declaration_was_marked_as_deprecated_here) would also be painful, because these messages are even less stable than the codes.

I really don't know if there is a good way to ignore specific errors :(

If the check is not explicitly requested for a particular test case, we perform a TSC check when one of services.getSymbolAtLocation/services.getTypeAtLocation or services.program.* is accessed

That ... could work somewhat. But it's a little complex. We'd have to wrap versions of the services used in tests. Plus, rules don't guarantee always calling to those APIs. There will likely be individual cases where they're not called, but type errors should still be checked.

Maybe this can just always be on if meta.docs.requiresTypeChecking is enabled? That would be simpler.

So this would be a kind of opt-out feature, right?

I'd think so, yeah.

My thought too. We would check for type errors if the rule requires type checking. And each test case can have an option to disable type error checking.

A few thoughts after iterating on #8351:

1. @ts-expect-error in the tests looks like a pretty handy approach to ignore TS errors in some specific places. That way we don't have to ignore TS errors for the entire test case, we can just surgically ignore errors in some specific line.
2. meta.docs.requiresTypeChecking is not a fixed key (like description or url), but a recommended name to tell eslint-doc-generator that the rule uses typed linting. We will have to mention this in RuleTester docs. Perhaps we could also allow customization of which docs key should be used to decide whether to collect TS diagnostics or not?
3. Do we need to collect TS diagnostics from fixture files or not? I'm leaning towards doing that as well. What do you think?

1. @ts-expect-error in the tests looks like a pretty handy approach to ignore TS errors in some specific places. That way we don't have to ignore TS errors for the entire test case, we can just surgically ignore errors in some specific line.

👍 agreed

2. meta.docs.requiresTypeChecking is not a fixed key (like description or url), but a recommended name to tell eslint-doc-generator that the rule uses typed linting. We will have to mention this in RuleTester docs. Perhaps we could also allow customization of which docs key should be used to decide whether to collect TS diagnostics or not?

Good point. That property has been kind of an unofficial standard for a while, including default support by https://github.com/bmish/eslint-doc-generator... but this would be the first instance of it being recognized by any other tooling.

My hunch is we'd want to have this be an opt-in option to start, to minimize how much of a breaking change it'd be. So folks could do new RuleTester({ typeCheckIndicator: 'requiresTypeChecking" }) or something like that? There's probably a better API name than that...

3. Do we need to collect TS diagnostics from fixture files or not? I'm leaning towards doing that as well. What do you think?

I like it. If it makes it a lot more complex I wouldn't be upset about it being a followup though.