Plugin for Search

[TheodoreChu]

I'm building an editor extension for Standard Notes called the Append Editor (Source). It makes heavy use of Remark and Rehype plugins to parse Markdown into React, so I'm very thankful for everything that the Remark and Rehype plugins already do.

I would like to build a feature to search within the editor, which is sometimes embedded within an iframe and does not have access to a browser's built-in search. I'm a new developer, so I don't have much experience building search inside applications.

I thought a potential solution would be to have a plugin for Remark, or probably Rehype, to allow for searching a keyword within the parsed/rendered html. For example, Remark and Rehype would parse the following Markdown text into the following html:

Markdown:

# Title

We want to search for a keyword within this text.

HTML:

<h1>Title</h1>
<p>We want to search for a keyword within this text.</p>

With this potential plugin, I am imagining that we would specify a keyword, such as keyword, and a class name, such as searchMatch and get something like this:

<h1>Title</h1>
<p>We want to search for a <span class="searchMatch">keyword</span> within this text.</p>

Is something like this possible and/or feasible with Remark/Rehype plugins? Would the plugin be more feasible as a Remark or Rehype plugin? Or is this outside the scope of what Unified tries to accomplish?

Thanks for taking the time to consider this.

[ChristianMurphy]

@TheodoreChu can you expand on two things:

1. what do you mean by "search"? (exact text matches, partial text matches, similarity scoring and ranking, semantic scoring and ranking)
2. what would you be searching? (a section of a document, a full document, many documents)

Broadly when I hear search I tend to think of Solr, ElastiSearch, Sphinx, MeiliSearch, etc. Full featured search engines, which have support for structured documents like Markdown or HTML, that have been processed with Unified.

Unified does have some single document search capabilities https://github.com/syntax-tree/nlcst-search for example. Which may work for some use-cases, but generally Unified is more focused on transforming a single document, than advanced cross document search.

[wooorm]

https://github.com/syntax-tree/hast-util-find-and-replace might be useful for searching with a regex and wrapping it in a span btw! (but it has some limitation, which Christian points out above)

[Nickersoft]

@wooorm I just stumbled across this thread after searching for a similar solution for my project – in particular, needing to match across HTML elements (as @ChristianMurphy mentioned in his first "trade off"). I'm curious how hard it would be to bring this functionality to hast-util-find-and-replace. I had been looking at mark.js, which implements this logic, but being new to Unified + not fully understanding the inner workings of mark.js, I'm a bit at a loss of how to bring it to Unified. I figured it's less expensive to load an HTML string into Unified than to load it into JSDOM just so I can use mark.js.

@TheodoreChu Were you ever able to come up with a solid solution to search?

[wooorm]

As I understand it, it combines all text nodes into a string, and saves information on positions in that string back to text nodes. It regexes the whole string, converts those places to original nodes, and marks each partial match. This can be done in hast as well.

[Nickersoft]

Thanks for such a quick response @wooorm! Yeah, after I posted my comment and dug a bit deeper I discovered that as well, which seems to be a reasonable approach. I think what I'm unsure of at this stage is converting between a DOM-based approach and the AST-based approach. For example, mark.js uses the built-in splitText() method for Text nodes, which I'm unsure how to reproduce in Unified. It also uses properties such as Node.previousSibling, which I'm not sure Unified has an alternative for.

Also, let me know if there's a more suitable place I should migrate this convo, such as a GitHub issue or separate discussion thread.

[wooorm]

The DOM is a complex model with lots of APIs that you need to know.
unist is objects, arrays, and strings. So you can use normal functions like slice to split a string or an array. To access a previous sibling in unist, you do parent.children[theIndex - 1].
The things you want to do, are also done by hast-util-find-and-replace, I think you’ll find most of what you’re looking for in its code.

As for where to ask things, that depends on where you want this conversation to go! If you think this can be added to find-and-replace, it could be there. If you want to make a new thing, a new discussion is probably better.