DSL support in Cursive with clojure.spec

[alex-dixon]

We might be able to get better IDE support. Potentially a lot better:
https://clojureverse.org/t/im-colin-fleming-ask-me-anything/1148/35

We'd need to use spec to define the DSL in order to for Cursive to be able to pick up on the grammar. From there I'm imagining defrules would be recognized as vars, and variable and result bindings may stop showing as unresolved variables. Maybe autocomplete/syntax checking would be a possible?

Cursive recently added support for some of Fulcro's macros in the latest Cursive EAP release. Overall it seems really enticing. If I understand correctly Colin wants to have IDE support for any DSL that is sufficiently defined with specs.

Since he mentioned Clara by name in the AMA I thought I'd raise the issue here and see if there's any interest in working with him on the feature, given it would require use of clojure.spec especially and I'm not sure where we stand on that.

[mrrodriguez]

@alex-dixon I think that sounds really interesting.
I actually messed around with spec'ing a DSL that was very similar to Clara's in the past, but that was long ago when spec was young.

This seems like it could be valuable to do an could perhaps even make the macros have better error reporting. The one thing I think that should be considered here is if it would be possible to somehow make the spec dependency (and associate clj 1.9) "optional".

Clara currently is supportive of older versions of clj/cljs and I think that could be considered valuable. However, I think the line of what versions are supported in both clj and cljs should still be considered as well as separate builds for different versions etc.

Do you have any thoughts on doing something like this that would be able to be "turned off" for older versions? I haven't thought about it much yet.

[alex-dixon]

Thanks. Great points.

I'll give it some thought and do some research, see what other people are doing etc. My gut reaction is 1. Supporting Clojure users previous to 1.9 would seem to necessitate some kind of schism and 2. That sounds like a nightmare! Don't do it!

If 1 entails 2, never use spec...?

I don't have enough experience to offer much more but will report back with anything I find. For what it's worth spec seems like it's here to stay, I think it addresses the "types" issue somewhat for people who like that in a language, and it provides an API for better stacktraces and error messages (one of the biggest complaints about Clojure) that people are already building upon in really exciting ways. e.g. Cursive just added "goto" support for specs, https://github.com/bhb/expound, https://github.com/jeaye/orchestra, https://github.com/stathissideris/spec-provider, test.check.generators, etc.

Might also consider other additions in 1.9 that could be valuable but also breaking (namespaced keywords?, namespaced maps?). Will look into that as well.

[cursive-ide]

In this case, I actually think supporting older versions is probably fairly easy. If you just want to use it to formalise the grammar and also for validation during macroexpansion, the specs can live in their own namespace and then that can be dynamically loaded somehow in Clojure 1.9+. Since I don't imagine that Clara will want to migrate to using spec to actually parse the macros (since that would be a major change to what I suspect is some pretty hairy code), then the two aspects can be kept separate (I believe, I'm no spec expert).

Edit: I asked over in #clojure-spec in Clojurians, and this seems to be easy. As an example it's what clojure.java.jdbc does. In fact, the specs could even be kept in a totally separate artifact if required.

[cursive-ide]

To clarify a little on how this would be useful for Cursive: Cursive doesn't actually use spec internally, it uses its own parsing system which predates spec and is more focused on macro parsing, not general data validation. I've recently rewritten the parser to semantically be closer to spec (it now uses regular expressions instead of PEGs), although that's not in Cursive proper yet. One of my goals for this was to be able to take macro specs and semi-automatically produce grammars for Cursive out of them. This is still only theoretical, but doing it for something non-trivial like Clara would really be helpful for me in figuring out how it should work. A lot of Cursive users are using Clara too so it would help them, and having specs will also potentially help any Clara users with the DSL too so it seems like a win-win. Even if the automatic translation doesn't work out, having the grammar specified would be a massive help in writing the Cursive grammars by hand, and of course Clara users can test the specs on a lot of code that I don't have access to - I can only write them based on public examples.

[alex-dixon]

Colin set up a repo to work on the specs! 🎉

https://github.com/cursive-ide/clara-specs

[mrrodriguez]

Colin set up a repo to work on the specs!

Indeed. I think these specs are looking pretty good too.

[abrooks]

I was looking to add clojure.spec to clara-rules both to displace the need for Prismatic schema (which breaks some self-hosting ClojureScript environments like Lumo) and for the added benefits of (A) better documenting the rule and query DSLs /AND/ removing much of the trickier code in https://github.com/cerner/clara-rules/blob/master/src/main/clojure/clara/rules/dsl.clj which simply becomes conform across the specs.

It looks like @cursive-ide already has much of the specs in place (maybe all of them, I didn't look closely enough). Is there anything blocking this? I'm interested in working on it if there's desire to pull clojure.specs in.

[rbrush]

Yes, I agree that we should include this in Clara proper and remove the old Prismatic schema. If you're interested in working on it, I'll be happy to support the effort and review and merge when ready. Thanks for looking into this!

I doubt we have too many users hitting the schemas so I doubt it will be disruptive, and we need to get on the evolutionary trunk of this stuff in Clojure.

[WilliamParker]

@abrooks +1 to getting spec support for things currently covered by Prismatic Schema. I think the reason we don't have it yet in Clara is just that nobody has had the time/interest in doing do. Spec does seem to be the way the community as a whole is moving and keeping up with that would be desirable as Ryan alludes to. If a PR is created I'll try to take at least a cursory look as well if I have time.

Do note that in addition to the DSL code and Prismatic schemas, there is custom validation code in the Clara compiler that deals with errors in the semantics of the rules (as opposed to their syntax). I haven't kept up with all the spec efforts by @cursive-ide but I don't see an obvious way to handle all of that logic with spec, and I don't think that was part of the effort (please correct me if I'm mistaken). Not that it would be a bad thing to do if possible.. just not sure that would be achievable. Just getting the level of validation done by Prismatic in spec would be very useful though.

[alex-dixon]

Pushed a start on this here:
cursive-ide/clara-specs@af65799

I think the main challenge is going to be figuring out how to support Clojure < 1.9 and use spec at the same time. So if anyone can figure that out we should be good. Converting from spec to schema should be extremely testable. I was able to set up a couple REPL tests and run data through a spec and the schema it's intended to cover pretty easily.

[alex-dixon]

Propose closing this if the main issue we wanted to deal with was IDE support. We could pick up spec and maybe > 1.9 support also in #403.