#136 Remove FollowSet cache dependency on ignoredTokens

[vityaman]

There is a problem explained at #136 (comment) because of cache dependency on ignoredTokens value. I just removed the token filter in getFollowingTokens. This logic originally was added in "Initial commit" 7 years ago, so motivation is unclear. On the one hand, tests were not changed, so the behavior with which all are agreed is the same. On the other hand, now we can get some elements of ignoredTokens in followSet from candidates. It can be fixed by simply adding a check after followSet was received either from the cache or computation.

@mike-lischke, there are two questions:

1. Do we want to filter tokens in follow set?

If yes, then I'll do changes and this test case will be OK:

core = CodeCompletionCore(X)
follow = core.collectCandidates(Y).tokens[Z] 
assert follow == [A, B, C]

core = CodeCompletionCore(X)
core.ignoredTokens.add(B)
follow = core.collectCandidates(Y).tokens[Z] 
assert follow == [A, C] // on 5eaad7b it returns [A, B, C]

2. Do we want to propagate this change to all ports (TypeScript, Java)?

[mike-lischke]

I see no reason why to remove a useful feature. OK, it's not a big thing, but it's something that makes usage a bit easier. The problem described in #136 seems not to be related to this filtering of unwanted tokens.

[vityaman]

@mike-lischke, returned following tokens filtering based on ignoredTokens. I placed it right after processRule inside collectCandidates to avoid caching dependent results.

[mike-lischke]

While I see the benefit of your changes, they will make it more difficult to incorporate future changes from the TS version (like bug fixes).

[vityaman]

@mike-lischke, if you are talking about this cache fix, I can make the same changes to Java and TypeScript versions of the library. Do we need it?

[mike-lischke]

No, no, that's not what I meant. I spoke about future changes in the TS version that would need to be tracked by the ports and the more the source code diverts, the harder it is to do that.