internal: Add comments to the lexer #4105
Conversation
This doesn't impact any external interfaces atm — it adds comments to the lexer and filters them out before they hit the parser. While we're still iterating on how comments (+ other significant whitespace) will be part of the formatter, we'll need comments in the lexer.
|
It would useful to have docblock comments in the AST so it could be used for documentation generation. |
Yes v good point |
Continuation of PRQL#4105
|
The lexer now creates a token for |
I agree but it might require some invasive work on the parser (check out the recent discussion about adding comments). Possibly we put this on the Statement struct and limit them doc comments to those... |
This seems reasonable - this would cover docs for variables (including functions), types, and modules. Is there anything else, we'd want to cover>? |
This doesn't impact any external interfaces atm — it adds comments to the lexer and filters them out before they hit the parser.
While we're still iterating on how comments (+ other significant whitespace) will be part of the formatter, we'll need comments in the lexer.