[ty] Fix diagnostics in ignored folders after adding new files#25236
Conversation
Typing conformance resultsNo changes detected ✅Current numbersThe percentage of diagnostics emitted that were expected errors held steady at 89.36%. The percentage of expected errors that received a diagnostic held steady at 85.49%. The number of fully passing files held steady at 88/134. |
Memory usage reportMemory usage unchanged ✅ |
|
Yeah, the high level API is too high and the low level API is too low. Sad. I've wanted a more ad hoc API as a middle layer for a while now. So in that sense, I'd love to have it upstreamed. But I'd really want the middle level API to be used as the implementation for the high level so that the very subtle, thorny and poorly coded logic isn't replicated. It can stay subtle, thorny and poorly coded, but limiting it to 1 copy would be nice. Whether I'll have the review bandwidth for that any time soon... IDK. I'd have to see the change. If it's cheap to produce a patch, it's worth a shot. Things that would make it easier:
I would be tempted to go the route of full re-traversal as a temporary fix depending on the aforementioned patch. Do you have a sense via a benchmarks of the cost of this? It's obviously slower, but how does it impact the UX? |
a32c503 to
9144c15
Compare
|
9144c15 to
08c3697
Compare
charliermarsh
left a comment
There was a problem hiding this comment.
This makes sense to me. I guess the "concerning" case would be something like: a Python file is created deep in the directory hierarchy, and there's no ignore at the root, and the root has a lot of files or directories in it, etc.?
Yes, the concern are directories that contain a large amount of files that are not ignored at the root level in combination with a burst of |
Summary
@dhruvmanila pointed out in #25183 (comment) that there are cases where ty incorrectly adds files to the project during an incremental walk if the created files (or folders) are within an ignored directory.
The root cause is that the
ignorecrate doesn't check whether an input path is ignored. It follows the order, you've told me to walk this directory so I will. And neither did ty. It simply walks any directory in which it sees new files.Unfortunately, there's no ad-hoc way of testing whether a directory or file is ignored. This is why, at least for now, this PR changes our incremental walking to always start from the project root, but incremental walking skips directories in which we didn't observe any newly created files or directories. However, we want to avoid unnecessary traversal when we can. E.g. we don't want to schedule a traversal for compilation results in the
targetdirectory if that directory is excluded. That's why this PR implements a heuristic for the common case where there is one root-levelignoreorgitignorefile. Any path that's excluded by those ignore files don't schedule a re-index.