Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve parsing for imports and exports #176

Merged
merged 4 commits into from
Jul 23, 2018
Merged

Improve parsing for imports and exports #176

merged 4 commits into from
Jul 23, 2018

Conversation

johno
Copy link
Member

@johno johno commented Jun 29, 2018

Before we relied 100% on remark's default parser which got us to an MVP but doesn't quite cut it due to the added syntax that imports, exports, and JSX. In many areas that syntax directly conflicts with MD syntax.

For example, consider the following export:

export const frontMatter = {
  heroImg: `${process.env.ASSET_PATH}/space-cats.png`,
  editUrl: 'https://github.com/mdx-js/mdx/edit/master/readme.md',
  authors: [
    'Wilma Flintstone',
    'Fred Flintstone'
  ]
}

The template string, url, and array conflict with Markdown parsing. This expected/intended behavior from an MD perspective, but not from an ES export. This occurs because the paragraph tokenizer then passes the parsed paragraph text on to inline tokenizers in order to parse links, inline code, emphasis, etc.

So we're now introducing our own block parsers that inject themselves before the HTML/paragraph block tokenizers that skip inline tokenization and pass along the raw values for imports/exports. Note that this is still a WIP

Also, I'd like to explicitly thank @wooorm for providing a wonderfully extensible library that allows us to do whacky things 💟.

@ChristopherBiscardi
Copy link
Member

It looks like this is something we'd need to support page queries in gatsby-mdx since right now the graphql tagged template literal is getting mangled as an inline-code element.

@johno johno changed the title [WIP] Improve ES/JSX parsing Improve ES/JSX parsing Jul 23, 2018
@johno johno changed the title Improve ES/JSX parsing Improve parsing for imports and exports Jul 23, 2018
Copy link
Member

@timneutkens timneutkens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@johno johno merged commit 583b3e5 into master Jul 23, 2018
@johno johno deleted the jsx-parsing branch July 23, 2018 14:54
@ChristopherBiscardi
Copy link
Member

This totally worked for us :D thanks!

johno added a commit that referenced this pull request Jul 25, 2018
* Spike out using custom tokenizer in remark parser

* Make import/export parsing more robust

* Revert unused change in mdxast test

* Move empty newline to utils
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants