Add org capture function

[Luis-Henriquez-Perez]

The code changes I have here aren't what I think should actually be added to consult--at least not raw. It's a starting point for adding an org capture function.

[minad]

Thank you. I take a look! Would it make sense to add these to org directly I wonder?

[Luis-Henriquez-Perez]

Would it make sense to add these to org directly I wonder?

All my pull requests have to do with org mode it would seem. Would you like me to close them and try to contribute them to org itself and open them if it doesn't go well? Or should I proceed with fine-tuning them for the inclusion into consult?

[minad]

@Luis-Henriquez-Perez Independent of where the functions will land in the end (consult or org), they must be finished and polished such that they can be packaged up.

The current status of this project is that up to now we mostly added functionality which is related to basic Emacs functionality and a few new functions like consult-line, which provides swiper for selectrum/icomplete-vertical. Furthermore there are functions like consult-multi-occur which improves on the Emacs multi-occur function but which should actually be part of Emacs proper. Furthermore @oantolin contributed consult-completion-in-region which is useful independent of the completion system and could as well go to Icomplete. But this way Icomplete stays a bit leaner and the function could prove useful if other completion systems are used.

Since we are not bound to a completion-system (in contrast to Counsel/Ivy), there is always the question in what we might want to add here. For Ivy there is no discussion basically, everything that should be enhanced by Ivy should go to Counsel. In our case we can add commands as well to Emacs proper or we can add them to Org directly. This is the beauty, the advantage but also the difficultiy of this design. We have the choice. The more upstream these functions are included the more useful they are since they are available to more users. If it is possible to add to Org I would go that route. In particular the #34 function seems to be a good candidate. There are however a few exceptions:

• You might want to use more completion-system-specific functionality or you might want to use the consult preview feature. Then the command should go here.
• You proposed Make consult-outline account for folding #21 and Preliminary code for unfolding org headline branch. #33, these seem to be independent of org-mode and rather related to general folding. The goal is to make consult work nicely with folded text.

Furthermore there are some contibution-process-related things to consider. Since org-mode is huge and widely used, maybe PRs take a long time to get into there? In that case one could consider adding useful functionality first here until it is added into upstream org. But then you could as well maintain the command inside your own init.el for the time being. What do you think - can you tell me more about that? Did you already submit things to Org at some point?

What I would not want: If a command is proposed for inclusion in Org, rejected there and then proposed for addition here. At least there must be very strong reasons in that case.

[oantolin]

This is off-topic, and I don't want to derail the conversion, but let me make a quick point about this:

Furthermore @oantolin contributed consult-completion-in-region which is useful independent of the completion system and could as well go to Icomplete.

I don't think that functions belongs in Icomplete, it could go in Emacs, but it wouldn't be in Icomplete, but rather in minibuffer.el. My point is that it isn't tied to any completion UI, and is not only useful for users of Icomplete. For example, people who use default tab completion abd the *Completions* buffer might want that function too. (Heck, I want that function and I only use icomplete when I'm fixing some problem in icomplete-vertical, the rest of the time I don't use icomplete at all.)

[minad]

I don't think that functions belongs in Icomplete, it could go in Emacs, but it wouldn't be in Icomplete, but rather in minibuffer.el.

Probably yes, but according to clemera it is quite common to have this or a similar function in completion systems packages. All I want to say - this function is very much completion-system related and for functions of that kind, consult aims to be a natural dumping group. This is contrast to the org functions, which might fit better into org directly.

@oantolin what do you think about the scope of consult? As you know from our discussions regarding embark, I am always looking a bit into how to structure packages or where functionality should be put such that it makes sense in the greater scheme of things.

[oantolin]

I don't think that function belongs in Icomplete, it could go in Emacs, but it wouldn't be in Icomplete, but rather in minibuffer.el.

Probably yes, but according to clemera it is quite common to have this or a similar function in completion systems packages.

I think people put this feature in completion packages mostly because they are not used to thinking about whether some functionality needs their package or can be built on completing-read alone. I hope this consult package of yours raises awareness of the adequacy of completing-read for many tasks.

All I want to say - this function is very much completion-system related and for functions of that kind, consult aims to be a natural dumping group. This is contrast to the org functions, which might fit better into org directly.

I agree!

@oantolin what do you think about the scope of consult? As you know from our discussions regarding embark, I am always looking a bit into how to structure packages or where functionality should be put such that it makes sense in the greater scheme of things.

Well, I think you should tell people what the scope of consult is, rather than asking for opinions. ;) But if you want my opinion, I'll be contributing to consult functions that are "general purpose" and use completion as an integral part of their UX but are independent of any particular completion UI.

The general purpose part is a little vague. I'm slightly on the fence about consult-org-capture actually. I think since org mode comes with Emacs, and org-capture is a widely used and fairly general part of Org, I'd be inclined to say a good consult-org-capture function probably does fit in with consult. I wouldn't add a consult-rhythmbox function though. :)

[minad]

Well, I think you should tell people what the scope of consult is, rather than asking for opinions.

Still finding out...

The general purpose part is a little vague. I'm slightly on the fence about consult-org-capture actually. I think since org mode comes with Emacs, and org-capture is a widely used and fairly general part of Org, I'd be inclined to say a good consult-org-capture function probably does fit in with consult.

I agree that org mode is important enough such that it could have its commands here. But I wonder what prevents these commands from being added to org directly. Maybe the issue is that many people already use these full-powered completion-systems which already bring their own commands, so the need to add something to org mode directly has never come up? When searching org.el, I find 22 matches for completing-read. So I don't think there is a barrier here.

I wouldn't add a consult-rhythmbox function though. :)

Damn, I was just writing that one! 🎶

[oantolin]

Well, the org-capture UI is pretty good, I bet most people like it and there just isn't much market for a completing-read version.

[Luis-Henriquez-Perez]

@minad

Since org-mode is huge and widely used, maybe PRs take a long time to get into there?

It would probably take a long time.

But then you could as well maintain the command inside your own init.el for the time being. What do you think - can you tell me more about that?

I have no problem maintaining them myself in my config. I suggested those commands here because I thought they would be useful to others (I certainly know I wished they had existed already when I wrote them--I mean they existed as counsel commands but I wished they existed generically using completing-read) and therefore would probably be better in one centralized place than independently implemented in configs.

Did you already submit things to Org at some point?

No, I haven't contributed to org before.

@oantolin

Well, the org-capture UI is pretty good, I bet most people like it and there just isn't much market for a completing-read version.

I can't speak for others, but in my individual case I was not satisfied with the default org UI. It's not that I found it lacking, but it was inconsistent with they way I completed everything else (via selectrum). Perhaps, I'm nit-picky but I want a consistent experience. So I certaintly am a part of that market (however small or large it may be)

[minad]

I think I am fine with having such commands here, but they need to be polished a bit and it would be great if you could write an explanation and a bit more background for each of the added commands, since my org knowledge is pretty basic unfortunately. For example regarding capturing, I agree with @oantolin that they ui provided by org is already pretty good. But still I see the appeal of having something based on minibuffer completion. Furthermore the command which sets the tags, org has the org-set-tags-command which seems pretty complicated when looking at the code in org.el. Now you replace that with a version based on completing-read-multiple, which is significantly simpler and I would like to hear why.

[Luis-Henriquez-Perez]

@oantolin
More thoughts in regard to the default org-capture interface:
It may very well be the case that it would be O.K. to use such an interface for certain selection. Maybe capture templates lend themselves more to that UI/hydra like selection as opposed to a vertical list like selectrum or ivy. In fact I rather like the org-capture UI. Still, it is unsatisfying to me that there is no standard for such an interface. As is often the case in org mode, these prompts are hard coded into the command (without even a variable to disable them--when I say this I'm thinking about my frustrating experience disabling the prompt from the selectrum-outline command) and they follow no standard. If I were to have these prompts for certain commands I would like to have one package behind all of them whether it be ivy-hydra, transient, or something else.

For curiosity's sake this is the recent personal experience that I thought of when I made my claim of Org UIs being hardcoded and having no standard:

I noticed that when I used this org/goto-headline command in my config, it would open this annoying popup--but only the first time it was called. After that it was fine. I spent a whole morning figuring out what caused the popup and how to disable it. I even instrumented the function, but the popup wouldn't show up if I did that! I realized somehow org-location was called just the first time it was called. In the end I gave up and mapped org location to call org/goto-headline.

(advice-add #'org-goto-location :override (lambda (&rest _) (call-interactively #'org/goto-headline)))

(defun org/goto-headline ()
  "Jump to an org heading using selectrum."
  (interactive)
  (require 'org-refile)
  (let ((selectrum-should-sort-p nil)
        (org-outline-path-complete-in-steps nil)
        (org-goto-interface 'outline-path-completion))
    (org-goto)))

[Luis-Henriquez-Perez]

@minad

org has the org-set-tags-command which seems pretty complicated when looking at the code in org.el. Now you replace that with a version based on completing-read-multiple, which is significantly simpler and I would like to hear why.

In general, I am not a fan of the way that org commands are written. From the code that I've seen in org-refile, org-babel-tangle-file, org-insert-heading and more, each command is packed with options. The control flow is extremely complex to the point where it is difficult to modify a command even slightly without accidentally changing part of the behavior, much less understand it. To make matters worse, condition clauses are often something terse like: (equal '(4) arg)--adding to legibility problems. Often the basic functionality and intent of the command is lost to the complexity of obscure, opinionated options (see org-insert-heading for an example). Believe it or not, org-set-tags-command is one of the simpler org commands comparatively speaking.

But to answer your question, all my command does is set the tags. It does not deal with the org-fast-tag-selection interface or org-loop-headlines-in-active-region. If you select a tag that is not already in the current heading, it adds it. If a tag you select already exists in the current heading, it removes it. Simple, useful, clear and to the point.

An example:
current heading tags -> :hi: :ho:
selected tags -> :hi: :he:
current heading tags -> :he: :ho:

I question whether desirable thing try to preserve all of the behavior of the org-set-tags-command. What are your thoughts on that?

[minad]

@Luis-Henriquez-Perez I agree with you that it is nice to have small commands which are easy to understand etc. You say that the org commands are complicated and confusingly written - I think many people dislike that. But it is not sufficient motivation to add commands here. The original org commands are broken beyond repair, we add a completely new UI now somewhere else?

A better motivation from my perspective is given for many of the consult commands already present, which either add something new (consult-line) or improve upon something already existing giving it new capabilities etc. For example org-capture has this hydra-like interface - then I think it is a good motivation to add something based on completing-read, since that might work better if you have many templates in contrast to the hydra interface? But as far as I understood, @oantolin would probably still continue to use the existing org interface? But then he says

I'd be inclined to say a good consult-org-capture function probably does fit in with consult.

For me the problem here is that my judgement regarding org is limited. I need a few days to try out a few things. In the meantime it would be nice if you could polish the capture and the tags command such that they adhere to the naming conventions etc and are ready for potential inclusion.

[oantolin]

Just to emphasize, you did correctly state my opinions: I do think it might be reasonable to have a completing-read based consult-org-capture in consult and at the same time, I do not think I would use it personally.

[Luis-Henriquez-Perez]

For example org-capture has this hydra-like interface - then I think it is a good motivation to add something based on completing-read,

Agreed.

The original org commands are broken beyond repair, we add a completely new UI now somewhere else?

They're usable if you don't care about details like whitespace, random popups, or all you windows suddenly being deleted. I'm one of the people that does care so I belive a simpler and more consistent org api is a worthy endevour that as you said is best reserved for it's own package.

For me the problem here is that my judgement regarding org is limited. I need a few days to try out a few things. In the meantime it would be nice if you could polish the capture and the tags command such that they adhere to the naming conventions etc and are ready for potential inclusion.

@minad will do. I'll get some work done on these today.

[minad]

@Luis-Henriquez-Perez If it makes sense to create another package to create an entirely new consistent org UI, that could as well go into its own package. If that is not the case and you end up with only a handful of commands they could as well go to consult. And then there is still the option to contribute to org itself, trying to improve things there. But as far as I understood this will be very hard if you don't want to break the existing behavior and given the intricate nature of the org commands.

[Luis-Henriquez-Perez]

I simplified the function quite a bit at the risk of breaking something. This function is based off of counsel-org-capture so the code wasn't all my own. But just by looking at the body I could tell there has to be some way to make this cleaner. I have no idea why it's doing everything it does in the function passed to mapcan. As long as org-capture templates have the keys as the first element and the description as the second I think this should work.

[minad]

@Luis-Henriquez-Perez Since we now have the consult-flycheck.el and consult-selectrum.el packages, I think the way to go for org is to create a consult-org.el package maintained within this repository. While org is an Emacs core component, I think making an extra package is justified if we have multiple consult-org-* functions, and from what I see this is the direction we are going?

[Luis-Henriquez-Perez]

@minad that sounds reasonable. So I'll move these functions to a separate file then.

[minad]

@Luis-Henriquez-Perez What is the status on your consult-org-* functions? I am closing this PR due to conflicts with the main/master branch. Please reopen the PR again rebased on the main branch!

[astoff]

I believe the correct approach to this would be to define a consult override to the org-mks function from org-macs.el. But I'm not sure all those proto-transient Org menus use this function. Maybe some are made by hand.

[astoff]

Now going on a slight tangent, in my case it's more useful to use consult to decide where the captured thing will land than to choose the type of capture, which is what I think this patch does. I just added the following to my config:

(add-to-list 'org-capture-templates
             '("c" "Consult..." entry (function consult-org-agenda)
               "* TODO %?\n  %i" :prepend t))

So when I capture something of category c, I first get to select an agenda heading under which the capture will be placed.

I also bind a key directly to this capture type, because I don't really need any other capture types:

(define-key blah "blah" (lambda () "Consult capture" (interactive) (org-capture nil "c")))

[jhenahan]

I'm currently using something like this:

(defun the--flipped-filter (coll pred)
  (seq-filter pred coll))

(defun the--org-capture-templates ()
  "Return a list of capture templates."
  (thread-first org-capture-templates
    ;; If you have nested capture templates (e.g., I have one at `pt` for a personal task and one at `wt` for a work task)
    ;; we'll list a pointless completion option, so filter out short items
    (the--flipped-filter (lambda (template) (> (length template) 2)))
    (org-capture-upgrade-templates)
    (org-contextualize-keys org-capture-templates-contexts)))

(defun the--lookup-org-capture-template (key templates)
  (seq-find (lambda (it)
              (equal (car it) key))
            templates))

(defun the--annotate-org-capture-templates (key templates)
  (concat
   "\t" (nth 1 (the--lookup-org-capture-template key templates))))

(defun the--org-capture ()
  (interactive)
  (let* ((templates (the--org-capture-templates))
         (annotate (lambda (cand) (the--annotate-org-capture-templates cand templates)))
         (selection (consult--read templates :prompt "Capture template: " :annotate annotate)))
    (org-capture nil selection)))

This at least gets me the name of the task, though I'd really like to figure out a nice way to be able to surface the "parent" key descriptions so I could have something like Work > Task in the annotation. I'm thinking of doing some more preprocessing on org-capture-templates to get it into a shape that's easier to work with for things like that, but I haven't gotten there just yet.

[astoff]

@jhenahan Did you try turning a version of this code into an advice to org-mks from org-macs.el, so you get this style of menus everywhere in Org mode?

[jhenahan]

I haven't, but I'll see if I can make that work. I'll do some reading. Might make it more obvious how to generalize this to get richer annotations, too.