fix(markdown-serializer): Override Turndown escaping behaviour with custom rules #102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for doist-typist ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
rfgamaral
force-pushed
the
ricardo/improve-escaping-mechanism
branch
from
1e4a581
to
bfdd2ca
Compare
rfgamaral
added
⛴ Ship PR
…ustom rules BREAKING CHANGE: Removed `options` parameter from `createMarkdownSerializer` function.
rfgamaral
force-pushed
the
ricardo/improve-escaping-mechanism
branch
from
bfdd2ca
to
7658862
Compare
|
A patch of this PR has been in production for at least 3 weeks now, and no major bugs have been reported. I believe this fix to be in an optimal state to be merged as-is, and will do just that once all statuses checks are green. |
doistbot
added a commit
that referenced
this pull request
Feb 23, 2023
## [1.0.17](v1.0.16...v1.0.17) (2023-02-23) ### Bug Fixes * **markdown-serializer:** Override Turndown escaping behaviour with custom rules ([#102](#102)) ([6950afb](6950afb))
|
🎉 This PR is included in version 1.0.17 🎉 The release is available on: Your semantic-release bot 📦🚀 |
2 tasks
jerrywcy
added a commit
to jerrywcy/memos-typist-editor
that referenced
this pull request
Aug 12, 2023
## 1.0.0 (2023-08-12) ### Features * Add the `PasteHTMLTableAsString` extension ([Doist#290](https://github.com/jerrywcy/memos-typist-editor/issues/290)) ([ee90014](ee90014)) * Allow all marks to coexist with the Code mark ([Doist#309](https://github.com/jerrywcy/memos-typist-editor/issues/309)) ([ac06735](ac06735)) * Disallow suggestions inside inline code marks and code blocks ([Doist#154](https://github.com/jerrywcy/memos-typist-editor/issues/154)) ([7d75314](7d75314)) * Initial Typist implementation ([19451ab](19451ab)) * **serializers:** Add `get*` functions for reusable singular instances ([Doist#88](https://github.com/jerrywcy/memos-typist-editor/issues/88)) ([b2c77c3](b2c77c3)) * TaskItem and TaskList extension ([5db1900](5db1900)) ### Bug Fixes * `insertMarkdownContent` didn't insert Markdown correctly in plain-text documents ([Doist#13](https://github.com/jerrywcy/memos-typist-editor/issues/13)) ([74cc623](74cc623)) * Add required ProseMirror dependencies to package ([Doist#73](https://github.com/jerrywcy/memos-typist-editor/issues/73)) ([cd605c0](cd605c0)) * Add support for literal autolinks (GFM based) ([Doist#303](https://github.com/jerrywcy/memos-typist-editor/issues/303)) ([4537091](4537091)) * Add support for query params in `RichTextLink` extension ([Doist#4](https://github.com/jerrywcy/memos-typist-editor/issues/4)) ([9fac158](9fac158)) * Bold and Italic now use *** and ___ instead of ** and __ ([cc29ea9](cc29ea9)) * change to memos-typist-editor in package.json ([c8250b1](c8250b1)) * **deps:** Migrate ProseMirror dependencies to `@tiptap/pm` package ([Doist#151](https://github.com/jerrywcy/memos-typist-editor/issues/151)) ([d2a8eae](d2a8eae)) * **deps:** update dependency prosemirror-codemark to v0.4.2 ([Doist#34](https://github.com/jerrywcy/memos-typist-editor/issues/34)) ([58938a1](58938a1)) * **deps:** update dependency prosemirror-model to v1.18.2 ([Doist#25](https://github.com/jerrywcy/memos-typist-editor/issues/25)) ([5d1fc1b](5d1fc1b)) * **deps:** update dependency prosemirror-model to v1.18.3 ([Doist#30](https://github.com/jerrywcy/memos-typist-editor/issues/30)) ([54bfd56](54bfd56)) * **deps:** update dependency prosemirror-view to v1.29.1 ([Doist#26](https://github.com/jerrywcy/memos-typist-editor/issues/26)) ([9f86a5e](9f86a5e)) * **deps:** update dependency unist-util-remove to v4 ([Doist#380](https://github.com/jerrywcy/memos-typist-editor/issues/380)) ([6c5430a](6c5430a)) * **deps:** update gfm autolink literal packages to v2 (major) ([Doist#378](https://github.com/jerrywcy/memos-typist-editor/issues/378)) ([7ad9af8](7ad9af8)) * **deps:** update gfm strikethrough packages to v2 (major) ([Doist#379](https://github.com/jerrywcy/memos-typist-editor/issues/379)) ([b1e411d](b1e411d)) * **deps:** update tiptap packages to v2.0.0 ([Doist#198](https://github.com/jerrywcy/memos-typist-editor/issues/198)) ([fe4aa82](fe4aa82)) * **deps:** update tiptap packages to v2.0.0-beta.200 ([Doist#3](https://github.com/jerrywcy/memos-typist-editor/issues/3)) ([6e977a9](6e977a9)) * **deps:** update tiptap packages to v2.0.0-beta.202 ([Doist#9](https://github.com/jerrywcy/memos-typist-editor/issues/9)) ([ce43f74](ce43f74)) * **deps:** update tiptap packages to v2.0.0-beta.203 ([Doist#35](https://github.com/jerrywcy/memos-typist-editor/issues/35)) ([2188bc6](2188bc6)) * **deps:** update tiptap packages to v2.0.0-beta.204 ([Doist#38](https://github.com/jerrywcy/memos-typist-editor/issues/38)) ([cb5b359](cb5b359)) * **deps:** update tiptap packages to v2.0.0-beta.205 ([Doist#54](https://github.com/jerrywcy/memos-typist-editor/issues/54)) ([2074402](2074402)) * **deps:** update tiptap packages to v2.0.0-beta.206 ([Doist#59](https://github.com/jerrywcy/memos-typist-editor/issues/59)) ([27e0f26](27e0f26)) * **deps:** update tiptap packages to v2.0.0-beta.207 ([Doist#63](https://github.com/jerrywcy/memos-typist-editor/issues/63)) ([da9889f](da9889f)) * **deps:** update tiptap packages to v2.0.0-beta.209 ([Doist#75](https://github.com/jerrywcy/memos-typist-editor/issues/75)) ([97a41c5](97a41c5)) * **deps:** update tiptap packages to v2.0.0-beta.215 ([Doist#129](https://github.com/jerrywcy/memos-typist-editor/issues/129)) ([af22d22](af22d22)) * **deps:** update tiptap packages to v2.0.0-beta.216 ([Doist#135](https://github.com/jerrywcy/memos-typist-editor/issues/135)) ([3d31a2e](3d31a2e)) * **deps:** update tiptap packages to v2.0.0-beta.217 ([Doist#136](https://github.com/jerrywcy/memos-typist-editor/issues/136)) ([eebba15](eebba15)) * **deps:** update tiptap packages to v2.0.0-beta.218 ([Doist#148](https://github.com/jerrywcy/memos-typist-editor/issues/148)) ([0e9e179](0e9e179)) * **deps:** update tiptap packages to v2.0.0-beta.220 ([Doist#158](https://github.com/jerrywcy/memos-typist-editor/issues/158)) ([c7fc3d8](c7fc3d8)) * **deps:** update tiptap packages to v2.0.0-rc.1 ([Doist#192](https://github.com/jerrywcy/memos-typist-editor/issues/192)) ([26d090a](26d090a)) * **deps:** update tiptap packages to v2.0.0-rc.2 ([Doist#195](https://github.com/jerrywcy/memos-typist-editor/issues/195)) ([8753f13](8753f13)) * **deps:** update tiptap packages to v2.0.1 ([Doist#201](https://github.com/jerrywcy/memos-typist-editor/issues/201)) ([e31cb2f](e31cb2f)) * **deps:** update tiptap packages to v2.0.2 ([Doist#212](https://github.com/jerrywcy/memos-typist-editor/issues/212)) ([200b339](200b339)) * **deps:** update tiptap packages to v2.0.3 ([Doist#226](https://github.com/jerrywcy/memos-typist-editor/issues/226)) ([a1953b0](a1953b0)) * **docs:** correct the render function name ([Doist#50](https://github.com/jerrywcy/memos-typist-editor/issues/50)) ([45dd681](45dd681)) * Extra paragraph node inserted above an Horizontal Rule ([Doist#313](https://github.com/jerrywcy/memos-typist-editor/issues/313)) ([3852309](3852309)) * **factories:** Allow alphanumeric IDs for suggestion nodes in `createSuggestionExtension` ([Doist#66](https://github.com/jerrywcy/memos-typist-editor/issues/66)) ([a1726a6](a1726a6)) * **html-serializer:** Disables tokenizers if marks/nodes are not found in the editor schema ([Doist#86](https://github.com/jerrywcy/memos-typist-editor/issues/86)) ([0ed4a9b](0ed4a9b)) * **html-serializer:** Don't share instances between editors ([Doist#275](https://github.com/jerrywcy/memos-typist-editor/issues/275)) ([3aba8c7](3aba8c7)) * Invalid `hasCodeMarkBefore` check in `canInsertSuggestion` utility function ([Doist#156](https://github.com/jerrywcy/memos-typist-editor/issues/156)) ([21826c5](21826c5)) * **markdown-serializer:** Override Turndown escaping behaviour with custom rules ([Doist#102](https://github.com/jerrywcy/memos-typist-editor/issues/102)) ([6950afb](6950afb)) * **paste-markdown:** Incorrect paste behaviour when HTML source is VSCode ([Doist#260](https://github.com/jerrywcy/memos-typist-editor/issues/260)) ([3326796](3326796)) * Remove unused Tippy.js peer dependency ([Doist#56](https://github.com/jerrywcy/memos-typist-editor/issues/56)) ([85f87a5](85f87a5)) * Replace usage of useEvent with useCallback ([Doist#98](https://github.com/jerrywcy/memos-typist-editor/issues/98)) ([3b175f7](3b175f7)) * **rich-text-link:** More lenient regex for input/paste rule ([Doist#72](https://github.com/jerrywcy/memos-typist-editor/issues/72)) ([98e363f](98e363f)) * Task List now only toggle by typing `[ ]` in bullet list ([ae3103b](ae3103b)) * Unit-tests now support TaskItem and TaskList ([639abf6](639abf6)) ### Reverts * Revert "ci: Add support to publish experimental releases" (Doist#279) ([dc57964](dc57964)), closes [Doist#279](https://github.com/jerrywcy/memos-typist-editor/issues/279) [Doist#278](https://github.com/jerrywcy/memos-typist-editor/issues/278)
jerrywcy
added a commit
to jerrywcy/memos-typist-editor
that referenced
this pull request
Aug 12, 2023
## 1.0.0 (2023-08-12) ### Features * Add the `PasteHTMLTableAsString` extension ([Doist#290](https://github.com/jerrywcy/memos-typist-editor/issues/290)) ([ee90014](ee90014)) * Allow all marks to coexist with the Code mark ([Doist#309](https://github.com/jerrywcy/memos-typist-editor/issues/309)) ([ac06735](ac06735)) * Disallow suggestions inside inline code marks and code blocks ([Doist#154](https://github.com/jerrywcy/memos-typist-editor/issues/154)) ([7d75314](7d75314)) * Initial Typist implementation ([19451ab](19451ab)) * **serializers:** Add `get*` functions for reusable singular instances ([Doist#88](https://github.com/jerrywcy/memos-typist-editor/issues/88)) ([b2c77c3](b2c77c3)) * TaskItem and TaskList extension ([5db1900](5db1900)) ### Bug Fixes * `insertMarkdownContent` didn't insert Markdown correctly in plain-text documents ([Doist#13](https://github.com/jerrywcy/memos-typist-editor/issues/13)) ([74cc623](74cc623)) * Add required ProseMirror dependencies to package ([Doist#73](https://github.com/jerrywcy/memos-typist-editor/issues/73)) ([cd605c0](cd605c0)) * Add support for literal autolinks (GFM based) ([Doist#303](https://github.com/jerrywcy/memos-typist-editor/issues/303)) ([4537091](4537091)) * Add support for query params in `RichTextLink` extension ([Doist#4](https://github.com/jerrywcy/memos-typist-editor/issues/4)) ([9fac158](9fac158)) * Bold and Italic now use *** and ___ instead of ** and __ ([cc29ea9](cc29ea9)) * change description in package.json ([52a2535](52a2535)) * change to memos-typist-editor in package.json ([c8250b1](c8250b1)) * **deps:** Migrate ProseMirror dependencies to `@tiptap/pm` package ([Doist#151](https://github.com/jerrywcy/memos-typist-editor/issues/151)) ([d2a8eae](d2a8eae)) * **deps:** update dependency prosemirror-codemark to v0.4.2 ([Doist#34](https://github.com/jerrywcy/memos-typist-editor/issues/34)) ([58938a1](58938a1)) * **deps:** update dependency prosemirror-model to v1.18.2 ([Doist#25](https://github.com/jerrywcy/memos-typist-editor/issues/25)) ([5d1fc1b](5d1fc1b)) * **deps:** update dependency prosemirror-model to v1.18.3 ([Doist#30](https://github.com/jerrywcy/memos-typist-editor/issues/30)) ([54bfd56](54bfd56)) * **deps:** update dependency prosemirror-view to v1.29.1 ([Doist#26](https://github.com/jerrywcy/memos-typist-editor/issues/26)) ([9f86a5e](9f86a5e)) * **deps:** update dependency unist-util-remove to v4 ([Doist#380](https://github.com/jerrywcy/memos-typist-editor/issues/380)) ([6c5430a](6c5430a)) * **deps:** update gfm autolink literal packages to v2 (major) ([Doist#378](https://github.com/jerrywcy/memos-typist-editor/issues/378)) ([7ad9af8](7ad9af8)) * **deps:** update gfm strikethrough packages to v2 (major) ([Doist#379](https://github.com/jerrywcy/memos-typist-editor/issues/379)) ([b1e411d](b1e411d)) * **deps:** update tiptap packages to v2.0.0 ([Doist#198](https://github.com/jerrywcy/memos-typist-editor/issues/198)) ([fe4aa82](fe4aa82)) * **deps:** update tiptap packages to v2.0.0-beta.200 ([Doist#3](https://github.com/jerrywcy/memos-typist-editor/issues/3)) ([6e977a9](6e977a9)) * **deps:** update tiptap packages to v2.0.0-beta.202 ([Doist#9](https://github.com/jerrywcy/memos-typist-editor/issues/9)) ([ce43f74](ce43f74)) * **deps:** update tiptap packages to v2.0.0-beta.203 ([Doist#35](https://github.com/jerrywcy/memos-typist-editor/issues/35)) ([2188bc6](2188bc6)) * **deps:** update tiptap packages to v2.0.0-beta.204 ([Doist#38](https://github.com/jerrywcy/memos-typist-editor/issues/38)) ([cb5b359](cb5b359)) * **deps:** update tiptap packages to v2.0.0-beta.205 ([Doist#54](https://github.com/jerrywcy/memos-typist-editor/issues/54)) ([2074402](2074402)) * **deps:** update tiptap packages to v2.0.0-beta.206 ([Doist#59](https://github.com/jerrywcy/memos-typist-editor/issues/59)) ([27e0f26](27e0f26)) * **deps:** update tiptap packages to v2.0.0-beta.207 ([Doist#63](https://github.com/jerrywcy/memos-typist-editor/issues/63)) ([da9889f](da9889f)) * **deps:** update tiptap packages to v2.0.0-beta.209 ([Doist#75](https://github.com/jerrywcy/memos-typist-editor/issues/75)) ([97a41c5](97a41c5)) * **deps:** update tiptap packages to v2.0.0-beta.215 ([Doist#129](https://github.com/jerrywcy/memos-typist-editor/issues/129)) ([af22d22](af22d22)) * **deps:** update tiptap packages to v2.0.0-beta.216 ([Doist#135](https://github.com/jerrywcy/memos-typist-editor/issues/135)) ([3d31a2e](3d31a2e)) * **deps:** update tiptap packages to v2.0.0-beta.217 ([Doist#136](https://github.com/jerrywcy/memos-typist-editor/issues/136)) ([eebba15](eebba15)) * **deps:** update tiptap packages to v2.0.0-beta.218 ([Doist#148](https://github.com/jerrywcy/memos-typist-editor/issues/148)) ([0e9e179](0e9e179)) * **deps:** update tiptap packages to v2.0.0-beta.220 ([Doist#158](https://github.com/jerrywcy/memos-typist-editor/issues/158)) ([c7fc3d8](c7fc3d8)) * **deps:** update tiptap packages to v2.0.0-rc.1 ([Doist#192](https://github.com/jerrywcy/memos-typist-editor/issues/192)) ([26d090a](26d090a)) * **deps:** update tiptap packages to v2.0.0-rc.2 ([Doist#195](https://github.com/jerrywcy/memos-typist-editor/issues/195)) ([8753f13](8753f13)) * **deps:** update tiptap packages to v2.0.1 ([Doist#201](https://github.com/jerrywcy/memos-typist-editor/issues/201)) ([e31cb2f](e31cb2f)) * **deps:** update tiptap packages to v2.0.2 ([Doist#212](https://github.com/jerrywcy/memos-typist-editor/issues/212)) ([200b339](200b339)) * **deps:** update tiptap packages to v2.0.3 ([Doist#226](https://github.com/jerrywcy/memos-typist-editor/issues/226)) ([a1953b0](a1953b0)) * **docs:** correct the render function name ([Doist#50](https://github.com/jerrywcy/memos-typist-editor/issues/50)) ([45dd681](45dd681)) * Extra paragraph node inserted above an Horizontal Rule ([Doist#313](https://github.com/jerrywcy/memos-typist-editor/issues/313)) ([3852309](3852309)) * **factories:** Allow alphanumeric IDs for suggestion nodes in `createSuggestionExtension` ([Doist#66](https://github.com/jerrywcy/memos-typist-editor/issues/66)) ([a1726a6](a1726a6)) * **html-serializer:** Disables tokenizers if marks/nodes are not found in the editor schema ([Doist#86](https://github.com/jerrywcy/memos-typist-editor/issues/86)) ([0ed4a9b](0ed4a9b)) * **html-serializer:** Don't share instances between editors ([Doist#275](https://github.com/jerrywcy/memos-typist-editor/issues/275)) ([3aba8c7](3aba8c7)) * Invalid `hasCodeMarkBefore` check in `canInsertSuggestion` utility function ([Doist#156](https://github.com/jerrywcy/memos-typist-editor/issues/156)) ([21826c5](21826c5)) * **markdown-serializer:** Override Turndown escaping behaviour with custom rules ([Doist#102](https://github.com/jerrywcy/memos-typist-editor/issues/102)) ([6950afb](6950afb)) * **paste-markdown:** Incorrect paste behaviour when HTML source is VSCode ([Doist#260](https://github.com/jerrywcy/memos-typist-editor/issues/260)) ([3326796](3326796)) * Remove unused Tippy.js peer dependency ([Doist#56](https://github.com/jerrywcy/memos-typist-editor/issues/56)) ([85f87a5](85f87a5)) * Replace usage of useEvent with useCallback ([Doist#98](https://github.com/jerrywcy/memos-typist-editor/issues/98)) ([3b175f7](3b175f7)) * **rich-text-link:** More lenient regex for input/paste rule ([Doist#72](https://github.com/jerrywcy/memos-typist-editor/issues/72)) ([98e363f](98e363f)) * Task List now only toggle by typing `[ ]` in bullet list ([ae3103b](ae3103b)) * Unit-tests now support TaskItem and TaskList ([639abf6](639abf6)) ### Reverts * Revert "ci: Add support to publish experimental releases" (Doist#279) ([dc57964](dc57964)), closes [Doist#279](https://github.com/jerrywcy/memos-typist-editor/issues/279) [Doist#278](https://github.com/jerrywcy/memos-typist-editor/issues/278)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
We've detected a few issues and inconsistencies with how our Markdown serializer escapes Markdown syntax, and this PR aims to fix that. The escaping is done by a bunch of aggressive built-in rules (based on regular expressions) implemented in Turndown, and this PR overrides the
escapefunction with a custom implementation with custom rules.We've been discussing this internally (ref), and this is the best solution I could come up with. However, I'm going to keep this in a draft state until we do a live test directly in production 🔥 (no better way to test this, really).
PR Checklist
Test plan
Since this will be tested directly in production, and given that unit tests have been written to make sure the Markdown serializer outputs exactly what's expected, a code-review should suffice.