Making CIDER more language agnostic? (decoupling Clojure assumptions from CIDER's nREPL client)

[Ruin0x11]

It feels like there is a gap for "interactive program exploration" that is only serviced very well in the Clojure ecosystem, like there now is for IDE features (Language Server Protocol) and debugging (Debug Adapter Protocol).

nREPL could be described as a language-agnostic spec, but in practice the implementations of nREPL clients almost always assume Clojure is being used. The documentation on the built-in operations and passed data types isn't as detailed as that for LSP, and the Clojure-specific operations are scattered throughout cider's source code without detailed documentation.

I have tried other nREPL clients but find they're missing features I want that cider does have, like module reloading, that I've implemented on the server side. I ended up having to write a totally separate REPL protocol and client code for one of my games to add the missing features. It worked leaps and bounds better than print debugging and I find that I can't go back to programming without it, but when I wanted to add it to a different project and hack in more program-specific features, I wondered why I couldn't just use nREPL as easily as I could as with my editor and an LSP server.

In contrast with LSP, which lets you pair any client with any server with the expectation that things will just work, you have to be careful not to choose an nREPL client/server that is missing features or assumes Clojure is being used.

What I would like is to be able to take cider and use it with an nREPL server in a language besides Clojure and not have to worry about incompatibilities. I understand this issue might be more of a standardization/onboarding issue than a programming one, but work on it could be done one step at a time. For example, some refactoring could be done on cider's nREPL client code to put Clojure-specific parts behind capabilities (as with LSP), such that cider will not blow up so long as the server doesn't specify that certain Clojure-specific features are available. And for earlier nREPL versions, cider could just fall back gracefully to the previous behavior and assume Clojure is being used, if it doesn't support the server capabilities feature.

Also, I have made an attempt in the past at writing a completely new protocol just for connecting to a remote REPL in a standardized, language-agnostic way, as with LSP and DAP, but found that doing so from scratch would take a very long time, and I wondered why not just leveraging nREPL was off-limits.

Would there any interest in making cider more generic across all programming languages?

[bbatsov]

That's one of my favourite topics these days, so let me add a couple of thoughts here.

nREPL could be described as a language-agnostic spec, but in practice the implementations of nREPL clients almost always assume Clojure is being used. The documentation on the built-in operations and passed data types isn't as detailed as that for LSP, and the Clojure-specific operations are scattered throughout cider's source code without detailed documentation.

Yeah, that's a pretty accurate description of the situation and obviously I'm to blame about the current state of affairs. When I started working on CIDER my only focus was implementing a Clojure programming environment, so I didn't really think much about the possibility of going beyond Clojure. Frankly, at the time I didn't even realize the potential of nREPL as language-agnostic protocol. Only when I eventually assumed the maintenance of nREPL a couple of years ago I saw the bigger picture and it dawned me that the nREPL protocol could be almost as important as LSP (at least for languages that are compatible with the overall idea).

Lately I've been spending more time documenting the nREPL protocol and promoting it as a Clojure-agnostic solution, but my progress has been fairly slow, as CIDER and nREPL obviously don't have the backing of Microsoft behind them. :-)

I have tried other nREPL clients but find they're missing features I want that cider does have, like module reloading, that I've implemented on the server side. I ended up having to write a totally separate REPL protocol and client code for one of my games to add the missing features. It worked leaps and bounds better than print debugging and I find that I can't go back to programming without it, but when I wanted to add it to a different project and hack in more program-specific features, I wondered why I couldn't just use nREPL as easily as I could as with my editor and an LSP server.

That is certainly my dream.

What I would like is to be able to take cider and use it with an nREPL server in a language besides Clojure and not have to worry about incompatibilities. I understand this issue might be more of a standardization/onboarding issue than a programming one, but work on it could be done one step at a time. For example, some refactoring could be done on cider's nREPL client code to put Clojure-specific parts behind capabilities (as with LSP), such that cider will not blow up so long as the server doesn't specify that certain Clojure-specific features are available. And for earlier nREPL versions, cider could just fall back gracefully to the previous behavior and assume Clojure is being used, if it doesn't support the server capabilities feature.

That's also something I've been working about. There's actually even an nREPL issue on the subject - nrepl/nrepl#181 We first need the nREPL servers to be able to communicate to the clients what's the runtime we're deal with, so the clients can act on this information. Alternatively we can have users specify explicitly those things (e.g. when doing cider-connect), but this doesn't seem to me like a great UX. In CIDER there are two big offenders to multi-platform support:

• Evaluation of some Clojure code here and there (something that's fairly easy to abstract away based on the server flavour). This is also coupled with the fact the results of evaluations are assumed to be Clojure/EDN and parsed as such.
• Functionality that's not part of the nREPL protocol and lives in cider-nrepl (e.g. debugging). Obviously alternative runtimes should be supporting those additional nREPL ops if they expect the client to work for them. I'm in the process if extending the nREPL support to have a native completion/lookup ops, but for many other things I'd wary to make them part of the protocol as most of them can't really be deemed "essential".

Would there any interest in making cider more generic across all programming languages?

Well, I'm not so sure about "all", as that's quite ambitious, but I'd definitely love to see CIDER supporting more programming languages than Clojure. Any help in this direction would be most welcome!

Recently, Conjure (nREPL client for vim) showed that nREPL is a viable path in this direction, as it now it supports both Clojure and Fennel and plans to add support for Racket as well.

[Ruin0x11]

Yeah, that's a pretty accurate description of the situation and obviously I'm to blame about the current state of affairs.

No worries, it was an unforseen issue. Thanks for everything you've done for CIDER and nREPL.

Here are various actionable issues I've been thinking about. Let me know if any of them sound effective.

• Improve the documentation for the current nREPL middleware, filling in missing return values and adding types for all parameters/return values. In particular it would really help if documentation for nested maps with the types of each field were available. This would make it considerably easier to program against the nREPL spec and add language support. I'm thinking that with the state of the protocol as is, we could consider each operation as a single "interface" to program against, which is language-agnostic. That would be so that no code has to be changed in CIDER to support different runtimes.
• Versioning the state of the nREPL spec alongside the nREPL library itself. This is so the protocol version can be specified correctly in the describe op.
• Add runtime information like in Log "foo" for (misc/log "foo"), not nil nrepl/nrepl#18, and special-case all Clojure-specific client code like directly evaling Clojure to only run if that runtime is being used. That way if the runtime is generic, no Clojure parts would be triggered. If there are any parts that require sending Clojure code from the client, they could be implemented as a new operation in the server instead.
• Adding versioning information about active middleware to describe, so that the middleware provided by CIDER can be versioned separately from nREPL's base operations.
• Write an nREPL server library for another language, like Lua or Ruby, and see if the spec can be implemented relatively cleanly.

[Ruin0x11]

For the moment I made an attempt at refactoring CIDER to support different runtimes, alongside related projects, to embed an nREPL written in Lua. So far it seems to work pretty well.

https://github.com/Ruin0x11/cider/tree/capability-gating
https://github.com/Ruin0x11/clj-refactor.el/tree/cider-capability-gating
https://gitlab.com/Ruin0x11/jeejah/-/tree/cider-enhance

[bbatsov]

Improve the documentation for the current nREPL middleware, filling in missing return values and adding types for all parameters/return values. In particular it would really help if documentation for nested maps with the types of each field were available. This would make it considerably easier to program against the nREPL spec and add language support. I'm thinking that with the state of the protocol as is, we could consider each operation as a single "interface" to program against, which is language-agnostic. That would be so that no code has to be changed in CIDER to support different runtimes.

No arguments from me.

Versioning the state of the nREPL spec alongside the nREPL library itself. This is so the protocol version can be specified correctly in the describe op.

That might be redundant if clients simply probe the capabilities of the server. There's also the question what versions to use for the protocol. Would they be the versions from the reference implementation or something different?

Add runtime information like in nrepl/nrepl#18, and special-case all Clojure-specific client code like directly evaling Clojure to only run if that runtime is being used. That way if the runtime is generic, no Clojure parts would be triggered. If there are any parts that require sending Clojure code from the client, they could be implemented as a new operation in the server instead.

Just a small remark - there's no such thing as generic runtime. :-) It's always some specific runtime.

If there are any parts that require sending Clojure code from the client, they could be implemented as a new operation in the server instead.

That's an overkill in many cases, so you'd rather just need to dispatch different code from the client for different runtimes.

Adding versioning information about active middleware to describe, so that the middleware provided by CIDER can be versioned separately from nREPL's base operations.

What would be the purpose of this? After all clients normally don't care where an op is coming from as long as they know how to handle it.

Write an nREPL server library for another language, like Lua or Ruby, and see if the spec can be implemented relatively cleanly.

I think we already have plenty of those, but judging from your next comment I think you saw this as well.