Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(TOML) Incorrect syntax highlighting #2594

Closed
aduh95 opened this issue Jun 8, 2020 · 0 comments · Fixed by #2595
Closed

(TOML) Incorrect syntax highlighting #2594

aduh95 opened this issue Jun 8, 2020 · 0 comments · Fixed by #2595
Labels
bug help welcome Could use help from community language

Comments

@aduh95
Copy link
Contributor

aduh95 commented Jun 8, 2020

Describe the issue

The library fail to parse some valid TOML lines.

Which language seems to have the issue?

TOML, also INI.

Are you using highlight or highlightAuto?

Both.

Sample Code to Reproduce

Using this code sample as test case (copied from TOML readme):

# Bare keys:
key = "value"
bare_key = "value"
bare-key = "value"
1234 = "value"

# Quoted keys:
"127.0.0.1" = "value"
"character encoding" = "value"
"ʎǝʞ" = "value"
'key2' = "value"
'quoted "value"' = "value"

"key \"containing\" backslash" = 6
'key \"containing" backslash\' = 6

# empty quoted key is allowed
"" = "blank"     # VALID but discouraged
'' = 'blank'     # VALID but discouraged

# Dotted keys:
name = "Orange"
physical.color = "orange"
physical.shape = "round"
site."google.com" = true
3.14159 = "pi"

# Whitespace around dot-separated parts is ignored:
hello  .  world  =  "!"

# Whitespace is ignored around key names and values
  hello = "World!"

JS code to render the above using highlight.js:

import hljs from "highlight.js/lib/core.js";
import toml from "highlight.js/lib/languages/ini.js";

const TOML = "toml";
hljs.registerLanguage(TOML, toml);

console.log(hljs.highlight(TOML,`
  …above toml code here…
`, true));

Actual behavior

Only bare keys and some dotted keys are highlighted, while the other ones are parsed as illegal.

<span class="hljs-comment"># Bare keys:</span>
<span class="hljs-attr">key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">bare_key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">bare-key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">1234</span> = <span class="hljs-string">"value"</span>

<span class="hljs-comment"># Quoted keys:</span>
"127.0.0.1" = "value"
"character encoding" = "value"
"ʎǝʞ" = "value"
'key2' = "value"
'quoted "value"' = "value"

"key "containing" backslash" = 6
'key "containing" backslash' = 6

<span class="hljs-comment"># empty quoted key is allowed</span>
"" = "blank"     <span class="hljs-comment"># VALID but discouraged</span>
'' = 'blank'     <span class="hljs-comment"># VALID but discouraged</span>

<span class="hljs-comment"># Dotted keys:</span>
<span class="hljs-attr">name</span> = <span class="hljs-string">"Orange"</span>
<span class="hljs-attr">physical.color</span> = <span class="hljs-string">"orange"</span>
<span class="hljs-attr">physical.shape</span> = <span class="hljs-string">"round"</span>
site."google.com" = true
<span class="hljs-attr">3.14159</span> = <span class="hljs-string">"pi"</span>

<span class="hljs-comment"># Whitespace around dot-separated parts is ignored:</span>
hello  .  world  =  "!"

<span class="hljs-comment"># Whitespace is ignored around key names and values</span>
    hello = "World!"

Expected behavior

The code sample contains valid TOML syntax, all lines should be highlighted.

<span class="hljs-comment"># Bare keys:</span>
<span class="hljs-attr">key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">bare_key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">bare-key</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">1234</span> = <span class="hljs-string">"value"</span>

<span class="hljs-comment"># Quoted keys:</span>
<span class="hljs-attr">"127.0.0.1"</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">"character encoding"</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">"ʎǝʞ"</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">'key2'</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">'quoted "value"'</span> = <span class="hljs-string">"value"</span>
<span class="hljs-attr">
"key "containing" backslash"</span> = <span class="hljs-number">6</span>
<span class="hljs-attr">'key "containing" backslash'</span> = <span class="hljs-number">6</span>

<span class="hljs-comment"># empty quoted key is allowed</span>
<span class="hljs-attr">""</span> = <span class="hljs-string">"blank"</span>     <span class="hljs-comment"># VALID but discouraged</span>
<span class="hljs-attr">''</span> = <span class="hljs-string">'blank'</span>     <span class="hljs-comment"># VALID but discouraged</span>

<span class="hljs-comment"># Dotted keys:</span>
<span class="hljs-attr">name</span> = <span class="hljs-string">"Orange"</span>
<span class="hljs-attr">physical.color</span> = <span class="hljs-string">"orange"</span>
<span class="hljs-attr">physical.shape</span> = <span class="hljs-string">"round"</span>
<span class="hljs-attr">site."google.com"</span> = <span class="hljs-literal">true</span>
<span class="hljs-attr">3.14159</span> = <span class="hljs-string">"pi"</span>

<span class="hljs-comment"># Whitespace around dot-separated parts is ignored:</span>
<span class="hljs-attr">hello  .  world</span>  =  <span class="hljs-string">"!"</span>

<span class="hljs-comment"># Whitespace is ignored around key names and values</span>
<span class="hljs-attr">    hello</span> = <span class="hljs-string">"World!"</span>
@aduh95 aduh95 added bug help welcome Could use help from community language labels Jun 8, 2020
aduh95 added a commit to aduh95/highlight.js that referenced this issue Jun 8, 2020
aduh95 added a commit to aduh95/highlight.js that referenced this issue Jun 8, 2020
aduh95 added a commit to aduh95/highlight.js that referenced this issue Jun 8, 2020
aduh95 added a commit to aduh95/highlight.js that referenced this issue Jun 9, 2020
aduh95 added a commit to aduh95/highlight.js that referenced this issue Jun 10, 2020
0xflotus added a commit to 0xflotus/highlight.js that referenced this issue Jun 12, 2020
* (docs) Add Chaos to supported languages (highlightjs#2510)

* fix(parser) fixes sublanguage with no rule matches (highlightjs#2506)

* fix(parser) fixes sublanguage with no rule matches

Resolves highlightjs#2504.

* (chore) Add ESLint config and clean up the major stuff (highlightjs#2503)

* (chore) eslint:recommended
* (chore): eslint_standard
* relax eslint rules for language grammars (to discourage rewriting them in one fell swoop; I'd rather have the blame history intact)
* remove extra escaping
* clean up variables
* more camelcase

* (docs) Add Visual Basic for Applications (VBA) to supported languages (highlightjs#2512)

* (yaml) improve tag support; add verbatim tags (highlightjs#2487)

* YAML parse non-word characters as part of tags
* adds support for verbatim tags

Co-authored-by: Josh Goebel <me@joshgoebel.com>

* fix(javascript/typescript): lambda with parens in parameters fails (highlightjs#2502)

* fix(javascript/typescript): lambda with parens in parameters fails

- Fixes both JavaScript and TypeScript grammars

Fixes samples like:

    const bad = ((a, b) => [...a, b]);
    sides.every((length,width=(3+2+(4/5))) => length > 0 );

This is done by counting parens in the regex that finds arrows
functions. Currently we can only handle 2 levels of nesting as
shown in the second example above.

* allow much richer highlighting inside params
* improve highlighting inside arguments on typescript

* enh(cpp): Improve highlighting of unterminated raw strings

PR highlightjs#1897 switched C++ raw strings to use backreferences, however this
breaks souce files where raw strings are truncated. Like comments, it
would be preferable to highlight them.

- Add `on:begin` and `on:end` to allow more granular matching when
  then end match is dynamic and based on a part of the begin match
- This deprecates the `endSameAsBegin` attribute. That attribute was
  a very specific way to solve this problem, but now we have a much
  more general solution in these added callbacks.

Also related: highlightjs#2259.

Co-authored-by: Josh Goebel <me@joshgoebel.com>

* (chore) C-like uses the new END_SAME_AS_BEGIN mode

* (chore) Ruby uses END_SAME_AS_BEGIN mode/rule

* (parser) make END_SAME_AS_BEGIN a function helper

Adds a mode helper to replace the deprecated `endSameAsBegin` attribute.

The first match group from the begin regex will be compared to the first
match group from the end regex and the end regex will only match if both
strings are identical.

Note this is more advanced functionality than before since now you can
match a larger selection of text yet only use a small portion of it for
the actual "end must match begin" portion.

* (pgsql) add test for $$ quoting existing behavior

- even if that existing behavior is questionable
- the ending span should really close before the $$, not after

Fixing this would involve delving into the sublanguage behavior and I'm
not sure we have time to tackle that right this moment.

* (chore) pgsql uses END_SAME_AS_BEGIN mode/rule now also

* (docs) rename to `mode_reference`; docs for callbacks

- I can never find this file because it's name didn't fully match.
- rename callbacks to `on:begin` and `on:end`

* prevented setter keyword conflicting with setTimeout|setInterval and highlighted them (highlightjs#2514) (highlightjs#2515)

* fix(javascript) prevent setter keyword 'set' conflicting with setTimeout|setInterval (highlightjs#2514)
* enh(javascript) setTimeout|setInterval now highlighted (highlightjs#2514)
* enh (javascript) clearInterval and clearTimeout now highlighted
* add keywords to TypeScript also

* (docs) add TLDR instructions for building and testing

* (dev) improve developer tool UI

* (parser) Build common EMCAscript foundation

Builds a common keyword foundation for any grammar that is
building on top of JavaScript:

- LiveScript
- CoffeeScript
- TypeScript

Also uses this common foundation for JS itself.

* (parser) Adds SHEBANG mode

* (yaml) Add support for inline sequences and mappings (highlightjs#2513)

* Use containers to match inline sequences and mappings
* Add string type for inside inline elements
* Handle nested inline sequences and mappings
* Disallow all braces brackets and commas from strings inside inline mappings or sequences
* clean up implementation
* feed the linter

Co-authored-by: Josh Goebel <me@joshgoebel.com>

* [enh] Add `OPTIMIZE:` and `HACK:` to comment doctags

* (build) browser build is CommonJS and IIFE, no more AMD (highlightjs#2511)

* (build) browser build is CommonJS and IIFE (global) now
* (build) dropping support for AMD, which we never truly supported
  properly in the first place
* (build) add test to make sure browser build works as commonJS module

  Resolves highlightjs#2505

* fix(parser) Fix freezing issue with illegal 0 width matches (highlightjs#2524)

* fix[parser] add edge case handle for illegal 0 width matches
* add last ditch catch all that tries to detect other uncaught freezes

* (docs) added unicorn-rails-log as an 3rd-party language (highlightjs#2528)

- (docs) Add syntax highlighting for Rails Unicorn logging to supported languages.

* (docs) fix supported languages link: it moved again! (highlightjs#2533)

* fix(ts/js) use identifier to match potential keywords (highlightjs#2519)

- (parser) Adds `keywords.$pattern` key to grammar definitions
- `lexemes` is now deprecated in favor of `keywords.$pattern` key
- enh(typescript) use identifier to match potential keywords, preventing false positives 
- enh(javascript) use identifier to match potential keywords, preventing false positives

* fix(javascript) fix regex inside parens after a non-regex (highlightjs#2531)

* make the object attr container smarter
* deal with multi-line comments also
* comments in any order, spanning multiple lines

Essentially makes the object attr container much more sensitive by allowing it to look-ahead thru comments to find object keys - and therefore prevent them from being incorrectly matched by the "value container" rule.

* (parser) Add hljs.registerAlias() public API (highlightjs#2540)

* Add hljs.registerAlias(alias, languageName) public API
* Add .registerAlias() test

* enh(cpp) add `pair`, `make_pair`, `priority_queue` as built-ins (highlightjs#2538)

* (fix) `fixMarkup` would rarely destroy markup when `useBR` was enabled (highlightjs#2532)

* enh(cpp) Recognize `priority_queue`, `pair` as containers (highlightjs#2541)

* chore: rename `registerAlias` to `registerAliases`

Plural form is clearly better as it's not surprising to the reader
to see it being passed an array - where as the singular form might
have been.  Meanwhile it's also easy to assume that it also supports
arrays of any size - including an array with a singular alias.

The fact that it can magically accept a string as the first argument
might not be obvious, but we document it and even if one didn't know
this they could still use the array form of the API without any issue
by passing a one item array.

* (swift) @objcMembers not completely highlighted (highlightjs#2543)

* Fixed @objcMembers in Swift

Would match `@objc` first, and the `Members` part would be unhighlighted

* Update CHANGES.md

* Update swift.js

* (docs) add OCL to list of supported languages (highlightjs#2547)

* (docs) Add Svelte to list of supported languages (highlightjs#2549)

* enh(dart) Add `late` and `required` keywords, and `Never` built-in type (Dart 2.9) (highlightjs#2551)

* Add new Dart 2.9 keywords for Null Safety language feature

* enh(erlang) add support for underscore separators in numeric literals (highlightjs#2554)

* (erlang) add support for underscore separators in numeric literals
* (erlang) add tests

* (docs) add Jolie to Supported Languages (highlightjs#2556)

* (parser/docs) Add jsdoc annotations and TypeScript type file (highlightjs#2517)

Adds JSDoc annotations and a .tsconfig that allows TypeScript to be run in it's "allowJS" mode and apply type and sanity checking to JavaScript code also. See Type Checking JavaScript Files.

I've been using TypeScript a lot lately and finding it very beneficial and wanted to get those same benefits here but without converting the whole project to TypeScript. It was rough at the beginning but now that this is finished I think it's about 80%-90% of the benefits without any of the TS compilation pipeline. The big difference in being JSDoc for adding typing information vs inline types with TypeScript.

Should be super helpful for maintainers using an editor with tight TypeScript integration and the improved docs/comments should help everyone else.

- Adds types/index.d.ts to NPM build (should be useful for TypeScript peeps)
- Improves documentation of many functions
- Adds JSDoc annotations to almost all functions
- Adds JSDoc type annotations to variables that can't be inferred
- Refactors a few smaller things to allow the TypeScript compiler to better infer what 
   is happening (and usually also made the code clearer)

* (parser) highlightBlock result key `re` => `relevance` (highlightjs#2553)

* enh(handlebars) Support for sub-expressions, path-expressions, hashes, block-parameters and literals (highlightjs#2344)

- `htmlbars` grammar is now deprecated. Use `handlebars` instead.

A stub is included so that anyone literally referencing the old `htmlbars` file (say manually requiring it in Node.js, etc) is still covered, but everyone should transition to `handlebars` now.

* fix(typescript) Add missing `readonly` keyword (highlightjs#2562)

* (docs) Mention `c` is a possible class for C (highlightjs#2577)

* fix(groovy) strings are not allowed inside ternary clauses (highlightjs#2565)

* fix(groovy) strings are not allowed inside ternary clauses
* whitespace can also include tabs

* Update @typescript-eslint/parser to the latest version 🚀 (highlightjs#2575)

* chore(package): update @typescript-eslint/parser to version 3.0.0
* chore(package): update lockfile package-lock.json

Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com>
Co-authored-by: Josh Goebel <me@joshgoebel.com>

* Update @typescript-eslint/eslint-plugin to the latest version 🚀 (highlightjs#2576)

* chore(package): update @typescript-eslint/eslint-plugin to version 3.0.0
* chore(package): update lockfile package-lock.json

Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com>
Co-authored-by: Josh Goebel <me@joshgoebel.com>

* (parser) properly escape ' and " in HTML output (highlightjs#2564)

* escape quotes also in final HTML output
* [style] update test coding style
* update markup tests with new escaping

This shouldn't be a security issue -- we've always escaped double quotes inside of HTML attribute values (where they could be used to break out of context) - and we've always used double quotes for enclosing attribute values. 

This just goes all the way and now properly escapes quotes everywhere.  Better safe than sorry.

* (docs) add changelog entry for last PR

* add nnfx theme (highlightjs#2571)

* (themes) Add new lioshi theme (highlightjs#2581)

* Added Cisco Command Line to SUPPORTED_LANGUAGES.md (highlightjs#2583)

* (themes) add `nnfx-dark` theme (highlightjs#2584)

* enh(protobuf) Support multiline comments  (highlightjs#2597)

* enh(java) added support for hexadecimal floating point literals (highlightjs#2509)

- Added support for many additional types of floating point literals
- Added related tests

There still may be a few gaps, but this is a pretty large improvement.

Co-authored-by: Josh Goebel <me@joshgoebel.com>

* (chore) Update issue templates (highlightjs#2574)

Co-authored-by: Vladimir Jimenez <allejo@me.com>

* enh(toml)(ini) Improve parsing of complex keys (highlightjs#2595)

Fixes: highlightjs#2594

* (chore) add `.js` extension to import statements (highlightjs#2601)

Adds file extensions to all import specifiers in ./src/ files.  This is useful to run the files straight from source with a web browser , Node.js ESM or Deno.

- Also add eslint rules regarding extensions for imports

* enh(dart) highlight built-in nullable types (highlightjs#2598)

* Dart: allow built-in nullable types with trailing ? to be highlighted

* enh(csharp) highlight generics in more cases (highlightjs#2599)

* (chore) fix tiny style issues, add linting npm task

- fixes tiny style issues
- adds `npm run lint` for linting the main library source
  (not languages which are still much messier)

* (chore) bump dev dependencies

* (chore) upgrade some dev stuff to newer versions

* bump v10.1.0

* (chore) bump copyright

* (chore) more import below metadata comment

Co-authored-by: M. Mert Yıldıran <mehmetmertyildiran@gmail.com>
Co-authored-by: Josh Goebel <me@joshgoebel.com>
Co-authored-by: Hugo Leblanc <dullin@hololink.org>
Co-authored-by: Peter Massey-Plantinga <plantinga.peter@gmail.com>
Co-authored-by: David Benjamin <davidben@google.com>
Co-authored-by: Vania Kucher <dev.kucher@gmail.com>
Co-authored-by: SweetPPro <sweetppro@users.noreply.github.com>
Co-authored-by: Alexandre ZANNI <16578570+noraj@users.noreply.github.com>
Co-authored-by: Taufik Nurrohman <t.nurrohman77@gmail.com>
Co-authored-by: Lin <50829219+Linhk1606@users.noreply.github.com>
Co-authored-by: nicked <nicked@gmail.com>
Co-authored-by: Nicolas Homble <nhomble@terpmail.umd.edu>
Co-authored-by: Ryandi Tjia <ryandi.tjia@me.com>
Co-authored-by: Sam Rawlins <srawlins@google.com>
Co-authored-by: Sergey Prokhorov <seriy.pr@gmail.com>
Co-authored-by: Brian Alberg <brian@alberg.org>
Co-authored-by: Nils Knappmeier <github@knappi.org>
Co-authored-by: Martin <7252614+Lhoerion@users.noreply.github.com>
Co-authored-by: Derek Lewis <DerekNonGeneric@inf.is>
Co-authored-by: greenkeeper[bot] <23040076+greenkeeper[bot]@users.noreply.github.com>
Co-authored-by: Jim Mason <jmason@ibinx.com>
Co-authored-by: lioshi <lionel.fenneteau@gmail.com>
Co-authored-by: BMatheas <65114274+BMatheas@users.noreply.github.com>
Co-authored-by: Pavel Evstigneev <pavel.evst@gmail.com>
Co-authored-by: Vladimir Jimenez <allejo@me.com>
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: TupikovVladimir <vladimir.tupikov@devexpress.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug help welcome Could use help from community language
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant