Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idea: use any comment block as test #49

Open
borkdude opened this issue Apr 7, 2022 · 22 comments
Open

Idea: use any comment block as test #49

borkdude opened this issue Apr 7, 2022 · 22 comments

Comments

@borkdude
Copy link
Contributor

borkdude commented Apr 7, 2022

It would be great if RCF could use any comment block as test, without having to add special syntax or a dependency for a library.

(ns my-lib)

(defn add [a b] (+ a b))

^:hyperfiddle/rcf
(comment
 (add 1 2) ;;=> 3
)

;;; rest of library namespace

This might require a different approach though and I could see that this would not be within RCF's scope. Just wanted to share the idea and see what you think of it. Feel free to migrate this issue to "Github Discussions" once you activate that in this repo, or close this issue.

@ieugen
Copy link
Contributor

ieugen commented Apr 7, 2022

Having comment blocks as test without having to require a library in the namespace would be huge IMO.
I have often thought of comment blocks as examples / documentation on how to use the functions and what kind of inputs it expects, even as tests.

But their usability as tests is limited that you have to run them one by one.
I would love to be able to have some tests there without having to pull in the test dependency in production builds.
I think this is where the tricky part lies: How to require things only in comments in a way that works with rcf and tooling.

rcf is a great initiative but I wouldn't want to pull in test libraries / dependencies in production builds.
Maybe it's just me but I think having a reduced surface area in production deployments is important.

@borkdude
Copy link
Contributor Author

borkdude commented Apr 7, 2022

A similar idea here:

https://github.com/lread/test-doc-blocks/blob/main/doc/01-user-guide.adoc#introduction

It takes your code sections from markdown and interprets:

(+ 1 2)
;;=> 3

as a test assertion.

@borkdude
Copy link
Contributor Author

borkdude commented Apr 7, 2022

I might do a little prototype of such a test runner using rewrite-clj.

@dustingetz
Copy link
Member

Thank you for sharing your idea! Parsing comment is tricky:

  • Async tests have additional syntax - rcf/% and rcf/!
  • ClojureScript support, for each way to build ClojureScript (Shadow comes with its own module loader)
  • Support for all possible ways to evaluate a test form
    • Support for naive "send buffer to REPL" editor commands in all editors
    • Support for sending a single test form to the REPL
    • Support for running tests in transitive dependencies when file is loaded at the REPL
  • Support for bypassing tests on REPL initial startup so REPL starts up as fast as possible
  • Support for CI
  • support for pattern matching and future syntax extensions like destructuring test results
  • support for custom test macros
  • support for deep assertions like (tests (let [x 1] (inc x) := 2))
  • support for syntax highlighting and structural editing
  • etc

There is also the question of is it moral, what are the risks, is it worth it? What would RH do?

@borkdude
Copy link
Contributor Author

borkdude commented Apr 7, 2022

Whether if it is moral, I think the question should be: should it be explicit? I think yes. That is why I put an explicit ^:hyperfiddle/rcf value on the comment form.

I just posted this idea for a lack of "Github Discussions", I think that is a much better fit, as this idea isn't really actionable.

@dustingetz
Copy link
Member

Ah, thank you for pointing out that you tag the comment for RCF evaluation, I missed that!

@dustingetz
Copy link
Member

dustingetz commented Apr 7, 2022

rcf is a great initiative but I wouldn't want to pull in test libraries / dependencies in production builds. Maybe it's just me but I think having a reduced surface area in production deployments is important.

Thank you @ieugen for the feedback! You can still separate your RCF tests into a tests folder, we merely remove the requirement. At Hyperfiddle we like to put 2-3 lightweight "pure function" tests next to the defn as example uses, and heavier test coverage and/or tests requiring setup and dependencies in separate files that only work under an alias. We're still debating if "heavy tests" need to be split out into a parallel folder (as that seems to be a historical requirement of legacy rest runners that watch the filesystem and rerun tests on file changes – which RCF's REPL-first design does not do. You can still separate a test folder if you want.)

@borkdude
Copy link
Contributor Author

borkdude commented Apr 7, 2022

@dustingetz Maybe going a bit off-topic here, but how does the infix notation play well with the REPL?

E.g.: (is (= 1 2)) you can eval in the REPL, but 1 := 2 not so much?

@dustingetz
Copy link
Member

(tests 1 := 2)

@borkdude
Copy link
Contributor Author

borkdude commented Apr 7, 2022

:-) But that's not how you normally work with rich comment forms?

@dustingetz
Copy link
Member

put your cursor on the 1 and send form to repl? I am confused

@ggeoffrey
Copy link
Member

For RCF to use any comment block as test, without having to add special syntax or a dependency, I assume we would have to hook onto the clojure reader. I’m not sure how, but if it’s possible it would be big.

If changing ;; => to => is acceptable, then data readers could work:

#rcf/tests
(comment
 (add 1 2) => 3
)

It saves a require and the #rcf/tests line can be commented out easily. Not sure if it’s a good tradeoff 🤔

@borkdude
Copy link
Contributor Author

borkdude commented Apr 8, 2022

@ggeoffrey I think you would have to go with an approach using rewrite-clj, or perhaps edamame or clojure.tools.reader, to read top level comments and process them differently.

See here for such an approach: #49 (comment)

@borkdude
Copy link
Contributor Author

borkdude commented Apr 8, 2022

Using data readers would still require you to register a data reader using a dependency (or no-op).

@ggeoffrey
Copy link
Member

Thank you for the link. It seems these approaches are unfortunately not compatible with a live REPL. As far as I can tell, the Clojure LispReader can’t be extended or overloaded. So it would force us into a completely different workflow (parsing files vs live coding).

@ggeoffrey
Copy link
Member

ggeoffrey commented Apr 8, 2022

:-) But that's not how you normally work with rich comment forms?

The idea is, in both of these forms what’s on right is like an annotation. We usually evaluate the left hand side at the REPL.

(comment
  1 ;; => 1
  )
(tests
  1 := 1)

IMO infix vs prefix should not be a constraint. Infix should just be sugar, meaning prefix could work too.

@borkdude
Copy link
Contributor Author

borkdude commented Apr 8, 2022

Makes sense now, thanks.

@borkdude
Copy link
Contributor Author

borkdude commented Apr 9, 2022

I think Stuart Halloway's transcriptor is a nice alternative approach:

https://github.com/cognitect-labs/transcriptor

(+ 1 2 3) ;; statement
(check! #{6}) ;; assertion

These appear in successive order and can be evaluated separately (to check if the assertion is valid during dev time for example).

@dustingetz
Copy link
Member

dustingetz commented Apr 9, 2022

Thanks for pointing that out – I don't think we considered that style. What do you think of

(require '[hyperfiddle.rcf :as rcf :refer [=> tests]])
(tests
  (map inc [1 2 3])
  (=> '(2 3 4)))

(tests (map inc [1 2 3]) (=> '(2 3 4)))

clojure.core/=>
Syntax error compiling at (REPL:0:0).
No such var: clojure.core/=>

@borkdude
Copy link
Contributor Author

borkdude commented Apr 9, 2022

Seems like an improvement to me!

matthewdowney added a commit to matthewdowney/rich-comment-tests that referenced this issue Dec 6, 2022
@borkdude
Copy link
Contributor Author

borkdude commented Jan 5, 2023

It seems https://github.com/matthewdowney/rich-comment-tests ran with this idea! :)

@marksto
Copy link

marksto commented Feb 25, 2024

It also worth noting that neither of these two brilliant libs (RCF nor RCT) eliminate the problem with dangling regular (uninstrumented) comment forms, that tend to "rot" over time because they are not called on frequently enough. An SCA tool, i.e. clj-kondo, is the only way to keep such comments up to date and workable. However, things go sour when you tag these forms with ^:clj-kondo/ignore to make REPL-based development easier. We've abandoned this practice and re-configured clj-kondo on one of our projects to prevent this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants