Change the return type of TokenList's prev/next to Option[Token]

[marcelocenerine]

The prev/next methods in TokenList return the input token if it is the head or last element of the tokenList:

val tokens = "".tokenize.get // this contains two tokens: BOF and EOF
val tokenList = TokenList(tokens) 

assert(tokenList.prev(tokens.head) == tokens.head)
assert(tokenList.next(tokens.last) == tokens.last)

In order to identify when a tokenList has run out of tokens while traversing it using prev or next, users must check whether the returned value is the same object passed as input, which is not a pretty standard thing in Scala libraries and can cause confusion. Maybe it would be clearer if those methods returned Option[Token] instead:

def next(token: Token): Option[Token]
def prev(token: Token): Option[Token]

[marcelocenerine]

I have this one implemented already but will wait until #534 (hopefully) gets merged.

[olafurpg]

@marcelocenerine I am afraid changing the signature is a breaking change that I'd like to avoid. I personally also like the current semantics since I use it regularly to do quick guards
like

+            case lf @ Token.LF() if ctx.tokenList.prev(lf).is[Token.Comma] =>

in #537 What do you think about adding instead nextOption/prevOption ?

[marcelocenerine]

@olafurpg that's true :(. If you want to avoid that and like the current semantics, then let's keep them as is.
If the api provided both next and nextOption, I think I would expect that next would throw if there is no next token (same for prev). Maybe we just add some basic scaladoc for now? Feel free to close this issue.

[olafurpg]

If the api provided both next and nextOption, I think I would expect that next would throw if there is no next token (same for prev).

I agree, I'm hesitant to add nextOption for this reason too.

I opened #539 to add a docstring to next/prev. Please take a look to see if the docstring can be clearer.