Implement hover and (stubbed) codeAction on stale state #5469

aprocter · 2022-03-14T23:13:45Z

Motivation

This implements a (beta, or let's even call it alpha) feature that allows hover queries to proceed even when slow-path typechecking is in progress. You have to opt into the feature with --enable-experimental-lsp-stale-state.

Test plan

See included automated tests.

main/lsp/LSPTypechecker.cc

jez · 2022-03-21T21:38:54Z

main/lsp/LSPTypechecker.h


 public:
-    LSPStaleTypechecker(UndoState &undoState) : undoState(undoState), pf(nullptr) {}
+    LSPStaleTypechecker(std::shared_ptr<const LSPConfiguration> config, UndoState &undoState);


It looks like there's an asymmetry between LSPStaleTypechecker and LSPTypecheckerDelegate here.

Why doesn't LSPTypecheckerDelegate take an LSPConfiguration or a WorkerPool?

The config is used inside LSPStaleTypechecker::query directly because LSPStaleTypechecker::query largely duplicates the implementation of LSPTypechecker::query, while LSPTypecheckerDelegate can just delegate LSPTypechecker::query, where config is a field on LSPTypechecker.

The WorkerPool we create here exists for similar reasons: LSPTypechecker has a private WorkerPool. (Note that here we create an empty, i.e., zero-thread, WorkerPool, which actually causes work to run on the indexing thread itself.)

jez · 2022-03-21T22:38:15Z

main/lsp/requests/hover.cc

+    if (typechecker.isStale()) {
+        config.logger->debug("HoverTask running on stale data");


Is there a way we can force the implementation of each LSPMethod to declare whether it's ok to be called with a stale state? I'd love to force opt-in, not opt-out.

(If that was already implemented in a previous PR I'd love to be pointed to the implementation.)

There's a virtual method on LSPTask that defaults to return false, and we'll only feed an LSPStaleTypechecker
if it returns true. Currently only hover and codeAction opt in.

sorbet/main/lsp/LSPTask.h

Lines 114 to 115 in db69c5c

// Returns true if the task can operate on typechecker stale state.

virtual bool canUseStaleData() const;

main/lsp/LSPTypechecker.h

aprocter · 2022-03-22T17:56:37Z

🚦 We're going to hold off on merging this til non-mutating namer/resolver are in place, so I've slapped a [WIP] on the title here.

Meanwhile I'm going to be working on getting some decent tests for this change.

aprocter · 2022-03-22T22:46:58Z

b9cd06d adds a simple test for a hover query in stale mode.

aprocter · 2022-03-29T17:00:45Z

Pulling this out of [WIP] because I think it's ready to go modulo namer/resolver updates. Since everything is hidden behind a flag, I think it'd actually be fine to merge this ahead of the namer/resolver updates, but I'm also fine with holding off until after those are ready.

Flipped a coin to pick a reviewer and jez was the lucky winner. :)

aprocter · 2022-03-29T17:04:43Z

I should also point out that this PR can only access indexed files that were saved into the UndoState when the slow path started running. A follow-on change (which I'm planning on opening as a stacked PR in a moment) will allow access to the rest of the indexed files. (Synchronization for that becomes pretty tricky, hence the separate PR.)

jez · 2022-03-29T17:08:29Z

test/lsp/multithreaded_protocol_test_corpus.cc

+    // Reset the options to enable stale state and sleep-in-slow-path.
+    auto opts = make_shared<realmain::options::Options>();
+    opts->lspStaleStateEnabled = true;
+    opts->sleepInSlowPath = true;


Is this required? This makes this test even slower than it currently is, right? (iiuc, this makes every slow path operation take 3 seconds artificially)

I believe a pause is the best option we have, but the slowness is honestly kind of nasty. (Let me get some numbers for you, but I believe that multithreaded_protocol_test takes substantially more time than on master.)

What we might be able to do is make the wait time configurable and set it to something much quicker than 3 seconds.

Is there no option to use something like what we do for other tests with the sorbet fence messages?

Yeah we're looking at:

master 32.18s PR 78.06s

So we'll need some solution here. Turning down the sleep time in runSlowPath might be a good one; maybe some sort of backdoor "sleep until the test harness tells you to wake up" is a better one, though I'm not sure how complicated that'd be off the top of my head.

Integrating the mechanism implemented in #5539 gives us:

master 32.18s #5539 34.16s (note: adds test of slow path blocking mechanism, which sleeps for 2s) PR 39.37s

Taking a 7-second hit here is still not wonderful, but I'm thinking it's probably acceptable since we're not planning on adding a huge pile of protocol tests anyway.

~~(Haven't pushed this to this branch yet, but will do so once #5539 hits master.)~~ done

froydnj · 2022-04-01T16:36:08Z

I think that since we now use non-mutating namer/resolver, this PR is probably ready for a proper review, is that right @aprocter ?

froydnj

This all looks reasonable; I think there are some type issues around pipeline::typecheck and getEvictedGs, but we can fix those up after the fact.

main/lsp/LSPTypechecker.cc

jez mentioned this pull request Mar 14, 2022

Namer pass that never mutates #5470

Closed

froydnj reviewed Mar 21, 2022

View reviewed changes

main/lsp/LSPTypechecker.cc Show resolved Hide resolved

aprocter force-pushed the aprocter/implement-stale-typechecker branch from f550e68 to 99cb8dd Compare March 21, 2022 21:15

jez reviewed Mar 21, 2022

View reviewed changes

aprocter changed the title ~~[WIP] Implement some actual non-stub functionality in the stale-state typechecker~~ Implement some actual non-stub functionality in the stale-state typechecker Mar 21, 2022

aprocter marked this pull request as ready for review March 21, 2022 22:18

aprocter requested a review from a team as a code owner March 21, 2022 22:18

aprocter requested review from elliottt and removed request for a team March 21, 2022 22:18

elliottt removed their request for review March 21, 2022 22:23

jez reviewed Mar 21, 2022

View reviewed changes

main/lsp/LSPTypechecker.h Show resolved Hide resolved

aprocter changed the title ~~Implement some actual non-stub functionality in the stale-state typechecker~~ [WIP] Implement some actual non-stub functionality in the stale-state typechecker Mar 22, 2022

jez mentioned this pull request Mar 25, 2022

Non-mutating version of namer #5498

Merged

aprocter force-pushed the aprocter/implement-stale-typechecker branch from a4b9dbb to a338c90 Compare March 29, 2022 16:56

aprocter changed the title ~~[WIP] Implement some actual non-stub functionality in the stale-state typechecker~~ Implement hover and (stubbed) codeAction on stale stae Mar 29, 2022

aprocter requested a review from jez March 29, 2022 16:58

jez reviewed Mar 29, 2022

View reviewed changes

aprocter changed the title ~~Implement hover and (stubbed) codeAction on stale stae~~ Implement hover and (stubbed) codeAction on stale state Mar 29, 2022

aprocter mentioned this pull request Mar 29, 2022

Expose more indexed files to LSPStaleTypechecker #5521

Closed

jez mentioned this pull request Mar 30, 2022

Non-mutating version of resolveConstants #5532

Merged

aprocter mentioned this pull request Mar 30, 2022

Add a mechanism to block the slow path in tests #5539

Merged

aprocter force-pushed the aprocter/implement-stale-typechecker branch from a338c90 to d8aa8f7 Compare March 30, 2022 18:49

aprocter mentioned this pull request Mar 31, 2022

Position tests for stale-state hovers #5545

Merged

aprocter force-pushed the aprocter/implement-stale-typechecker branch from 1fee8c8 to 15fd3ff Compare April 1, 2022 00:32

aprocter added 5 commits April 1, 2022 09:52

wip

855cc5c

wip

84813e6

Use slow-path blocking in the protocol tests

013415f

Remove some no-longer-needed plumbing

c17de4b

Switch to non-mutating resolver

2d90865

aprocter force-pushed the aprocter/implement-stale-typechecker branch from af3f14a to 2d90865 Compare April 1, 2022 16:52

froydnj approved these changes Apr 1, 2022

View reviewed changes

main/lsp/LSPTypechecker.cc Show resolved Hide resolved

aprocter merged commit a516eaf into master Apr 1, 2022

aprocter deleted the aprocter/implement-stale-typechecker branch April 1, 2022 19:02

		if (typechecker.isStale()) {
		config.logger->debug("HoverTask running on stale data");

	// Returns true if the task can operate on typechecker stale state.
	virtual bool canUseStaleData() const;

Implement hover and (stubbed) codeAction on stale state #5469

Implement hover and (stubbed) codeAction on stale state #5469

Uh oh!

Conversation

aprocter commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Test plan

Uh oh!

Uh oh!

jez Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

aprocter Mar 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jez Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

aprocter Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aprocter commented Mar 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aprocter commented Mar 22, 2022

Uh oh!

aprocter commented Mar 29, 2022

Uh oh!

aprocter commented Mar 29, 2022

Uh oh!

jez Mar 29, 2022

Choose a reason for hiding this comment

Uh oh!

aprocter Mar 29, 2022

Choose a reason for hiding this comment

Uh oh!

jez Mar 29, 2022

Choose a reason for hiding this comment

Uh oh!

aprocter Mar 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aprocter Mar 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

froydnj commented Apr 1, 2022

Uh oh!

froydnj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aprocter commented Mar 14, 2022 •

edited

Loading

aprocter Mar 21, 2022 •

edited

Loading

aprocter commented Mar 22, 2022 •

edited

Loading

aprocter Mar 29, 2022 •

edited

Loading

aprocter Mar 30, 2022 •

edited

Loading