Semantic tokens can still become desynced/offset

[0dinD]

There have been numerous issues about desynced semantic tokens in the past, but I thought I had found the root cause last time around. As it turns out however, there are two different kinds of token desynchronization.

The first problem, where there's a flicker of desync (referred to as "part two" in my comment), was fixed with eclipse-jdtls/eclipse.jdt.ls#1806 and #2000.

The second problem, where the tokens become permanently desynced until the next edit (referred to as "part one" in my comment), was partially fixed in eclipse-jdtls/eclipse.jdt.ls#1552. But there are still cases where this issue appears, so the root cause hasn't been found.

This issue does not appear to be a regression, as I've seen it happen in pretty much every version of the extension. The problem is just that it's so rare, that I could never find a reliable way to reproduce it (and I was also hoping it would get fixed with eclipse-jdtls/eclipse.jdt.ls#1806 and #2000).

Finally, by sheer luck, I managed to find some ideal conditions for reproducing it on my machine: so I recorded it (see below) and started investigating the root cause.

semantic-tokens-desync.mp4

Environment

• Java extension version: 1.0.0

Steps To Reproduce

1. Make multiple edits to a document, in quick succession. The success rate of reproducing the issue seems to be dependent on many factors: the performance of your machine (I'm reproducing it on a weaker laptop), the size of the document, etc.
2. If you're unlucky (it's quite rare and very random), one sequence of edits may result in desynced semantic tokens.
3. Only after making another edit do the tokens "repair" themselves.

Also see eclipse-jdtls/eclipse.jdt.ls#1918 for how to reproduce this with 100% accuracy using the debugger.

Additional Information

I will submit a separate issue in jdt.ls, as I believe to have found the root cause, which may affect more than just semantic tokens.

[DerDunkel]

Is there any short term solution for this?

[0dinD]

Is there any short term solution for this?

We are testing some workarounds to partially fix this issue (as you can see in the linked pull requests above). This might be released as a temporary solution. But in order to fully fix the issue, we probably need to investigate and fix the race condition in the upstream Eclipse JDT Core library, which is a bit more complex.