feature: Create UnindexedProject notification to be sent to the client

[davidbarsky]

(Note that this branch contains commits from #15830, which I'll rebase atop of as needed.)

Based on the discussion in #15837, I've added a notification and off-by-default toggle to send that notification from handle_did_open_text_document. I'm happy to rename/tweak this as needed.

I've been using this for a little bit, and it does seem to cause a little bit more indexing/work in rust-analyzer, but it's something that I'll profile as needed, I think.

[davidbarsky]

Did some profiling of this on my Mac and the profile below is generally representative:

   15ms - GlobalState::handle_event
       13ms - handle_did_open_text_document
           13ms - run_unindexed_project
               13ms - GlobalState::process_changes
                    1ms - RootDatabase::apply_change
                        1ms - RootDatabase::apply_change
                        0   - RootDatabase::request_cancellation (1 calls)
                        0   - ???
                   11ms - ???
                0   - relevant_crates (1 calls)
                0   - ???
            0   - ???
        0   - GlobalState::process_changes (1 calls)
        1ms - ???

13ms doesn't feel like the best, but is probably acceptable for an opt-in toggle.

[davidbarsky]

CI failed because of a timeout in receiving a notification, which doesn't feel great.

[Veykril]

Blocking in a notification handler is gonna end in a bad experience as notifications are handled on the main thread which means blocking there blocks processing anything else aside from thread pool tasks. It would probably make more sense to add a new Task kind for this and spawn the special handler on the task pool.

[davidbarsky]

Moved over to a task-based approach. I'm not fully sure if this is the approach that you had in mind since GlobalState::process_changes isn't running on a different thread, but I guess no other caller of it really is either.

[Veykril]

Now we are still processing it on the main thread :D All of this should be done in the spawned task on the thread pool, where as Task::FileIndexState should be used to return the result of the computation here (the snap.analysis.crates_for(id) result) and then have the task handler here just send the notification.

[davidbarsky]

After chatting about this over DMs, I'm not going to take the suggested approach and make this noop task more substantial since there is no way to ensure that this code runs after GlobalState::process_changes() has been called without doing one of the following:

• Add a retry mechanism for tasks, which would make this a de-facto spin loop. This might be okay!
• Creating a mechanism of enqueuing a task, but deferring spawning them until after a turn of GlobalState::handle_event. This ensures that GlobalState::process_changes() has been called.

[davidbarsky]

Right, I made two sets of changes that I would be completely happy with backing out on:

• I made a background task queue with a lightweight struct wrapper. Its only purpose is spawn a task after GlobalState::process_changes.
• To avoid adding a regular Task whose only purpose is to send a notification to the client, I opted to make lsp_server::ReqQueue implement Clone and I wrapped in an Arc and call Arc::make_mut as appropriate. This was necessary because the spawned task would move self across threads.
  • As an alternative, I can inline all two lines of GlobalState::send_notification into GlobalState::handle_queued_task and undo the changes made to lsp_server::ReqQueue and any subsequent knock-on changes. This means that if a QueuedTask ever grows an enum variant that needs to send a request, they might need to take the approach that I currently took with ReqQueue, but that day may never come—after all, the current approach to concurrency in rust-analyzer worked well until now.

[davidbarsky]

(Squashed commits; rebased against master. No meaningful change.)

[davidbarsky]

I've decided to back out of changes where I factored the GlobalState.sender and GlobalState.req_queue fields into a dedicated Conn struct alongside all the other, supporting changes (pervasive use of Arc::make_mut, cloneable fields in lsp_server). I've instead inlined the contents of GlobalState::send_notification into the event handler, which feels fine to me.

Anyways, I've had this PR deployed in my employer's buck2-based monorepo and it has driven down instance of rust-analyzer not running for folks dramatically.

[bors]

☔ The latest upstream changes (presumably #16182) made this pull request unmergeable. Please resolve the merge conflicts.

[Veykril]

There is a similar thing I added for whether proc-macros have changed in #16247, we might be able to switch that to this approach as well. Will need to check

[davidbarsky]

There is a similar thing I added for whether proc-macros have changed in #16247, we might be able to switch that to this approach as well. Will need to check

Having briefly looked over #16247, it's certainly a maybe. I considered using the request queue system, but that only really is able to handle a single event at a time. The task-based approach I did here can handle multiple events, which can happen with both FileDidOpenNotifications and proc macros being changed.

[bors]

☔ The latest upstream changes (presumably #16470) made this pull request unmergeable. Please resolve the merge conflicts.

[Veykril]

clippy is complaining on CI

[Veykril]

@bors delegate+

[bors]

✌️ @davidbarsky, you can now approve this pull request!

If @Veykril told you to "r=me" after making some further change, please make that change, then do @bors r=@Veykril

[davidbarsky]

@bors r=@Veykril

[bors]

📌 Commit 6330b02 has been approved by Veykril

It is now in the queue for this repository.

[bors]

⌛ Testing commit 6330b02 with merge 1370784...

[bors]

☀️ Test successful - checks-actions
Approved by: Veykril
Pushing 1370784 to master...