MarkdownQuery: mdq

A local text file Xapian indexer and query tool. Started life as a minimal Zettelkasten-like Markdown+FrontMatter document indexer and query interface: Keep lots of small notes and then quickly find them again using Information Retrival-style natural language queries.

I would like to extend this to index other bodies of text on a local filesystem, such as man/info pages. TODO

Example

For demonstration purposes, here I indexed a bunch of articles from Wikipedia.

About

How do you do personal notes? Grepping through a folder full of notes isn't the most effective approach; using Xapian is a better way.

I originally implemented this as a Vim plugin using some Python code built around Xapian, called tika. I decided to rewrite this in Rust, and after exploring other options for Information Retrieval based search tools like Tantivy and Fuzzy-Finder, I went back to Xapian: it's the best.

Note

This is my first Rust project, I welcome any comments/PRs/issues related to improving the code here.

There's still a lot I'd like to do here, but this version should at least be usable.

Wishlist of TODOs

• fix preview output
• use pest for query parsing
• Colorize preview
• Support platforms besides linux, mac
• Delete entries from local cache
• Query/filter on time range
• Vim Plugin
• TUI start list at the bottom instead of the top
• TUI select many
• searching/jumping-to/highlighting in preview
• pageup/down; ctrl-w
• Cache MD5 hashes of files using kv to skip indexing unchanged files
• Keep track of document access count in KV and use that as a weighting factor in query results
• Keep track of all Tags to be used for autocompletion
• cleanups, refactoring, rust-analyze, clippy and linting
• Optimize binary and compile times
  • https://matklad.github.io//2021/09/04/fast-rust-builds.html
  • https://pingcap.com/blog/rust-compilation-model-calamity
  • https://doc.rust-lang.org/rustc/profile-guided-optimization.html
• Prune dependencies: https://pingcap.com/blog/rust-compilation-model-calamity#recent-work-on-rust-compile-times
• CLI option for passing in starting query for interactive mode
• CLI option to emit JSON instead of filename
• import man/info pages and other canonical documentation for indexing and IR
• Add URL tag, support multiple?
• Support multiple Author tags
• Add tests
• Query/filter on tags (make sure this is working properly)
• statically link xapian-core

Installation

For now, clone this repo and run the makefile.

Usage

# Index a source directory, note the single quotes used here to prevent the
# shell from expanding the wildcard here
mdq [db dir] update '/path/to/markdown-directory'

# Run an interactive query against an index
mdq [db dir]

Note on Markdown+Frontmatter format

I would like to make this pluggable, but for now it's hardcoded to look for Markdown+Frontmatter files like this:

---
author: Steve Sosik
date: 2021-01-15T08:23:24-05:00
tags:
- vim
title: How to grep open buffers in Vim
---

Run this command

```
:bufdo vimgrepadd [search] % | copen
```

Requirements

lightly patched version of xapian-rusty, included here as a submodule.

zlib and xapian-core, which are bundled here.

To get started in a ubuntu-18.04 docker image:

apt-get update --yes
apt-get upgrade --yes
apt-get install --yes build-essential git
git clone --recurse-submodules https://github.com/ssosik/xapiary.git
cd xapiary
make

Development

Any modern standard Rust installation should probably work.

I use NixOS along with Direnv and Direnv Nix Integration

git clone --recurse-submodules git@github.com:ssosik/xapiary.git
cd xapiary
direnv allow
# Wait some time for Nix to install all of the Rust tooling
rustup install stable

make test
make run

# hack, hack, hack