Move collecting up tokens for a symbol to RubyLex #462
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RDoc::Markup::ToHtml
usesRDoc::RubyLex.tokenize
directly and callsRDoc::TokenStream.to_html
with returned tokens.On the one hand,
RDoc::Parser::RubyTools#get_tk
collects upTkSYMBEG
and trailing tokens inside the symbol with tokens byRDoc::RubyLex
.I moved logic what collects up tokens for a symbol from
RDoc::Parser::RubyTools#get_tk
toRDoc::RubyLex
. SoRDoc::TokenStream.to_html
becomes to handle symbols as value.Before, each symbols are separeted
TkSYMBEG
and trailing tokens:I think what
:
andhoge
withruby-identifier
attribute are separeted is not correct because symbol is one literal and one value. It's not identifier.After, each symbols are treated as literal with
ruby-value
attribute.