Genie - Run Clojure scripts with a daemon process

Introduction

Another way of mitigating the slow startup of Clojure on the JVM, by starting a daemon in the background. Using the following projects:

• nRepl - for running the daemon and providing client connections (using bencode).
• Pomegranate - for dynamically loading libraries.
• Babashka - for the client and installation script.

Why Genie?

From https://www.wordnik.com/words/genie:

• noun - A supernatural creature who does one’s bidding when summoned.
• noun - A fictional magical being that is typically bound to obey the commands of a mortal possessing its container.

So the genie in this case is the daemon process containing the nRepl server, waiting to execute Clojure scripts when summoned. It might help to see the daemon as part of the Operating System.

Goals

• Every script that can be executed by clj (using deps.edn) should also be able to be executed using Genie. And therefore you can also develop your script using e.g. CIDER.
• Fast startup of scripts.
• Dynamic loading of libraries.
• Performance similar to Clojure itself, as it is still Clojure on the JVM.
• Cross-platform: supports Linux, Windows and MacOS.
• Fast changing and running of scripts. Either develop in a (CIDER) repl, or by changing and running the script.
• Run multiple scripts simultaneously in one JVM.

Non-goals:

• The fastest possible startup - running with a client/server setup implies some (local) network overhead.
• Supporting long-running, full applications. Use default Clojure if you need this.

Installation

Prerequisites

• Java (>= v11)
• Clojure (>= v10)
• Genie uses Babashka for the client and installation. So make sure you have this installed.
• Leiningen is used for creating an uberjar. Using an uberjar will reduce daemon startup time.

Install with Babashka

bb ./install.clj

Install directories may be given, and some other options:

$ bb ./install.clj -h
install.clj - Babashka script to install Genie:
  daemon, client, config, scripts and template
      --daemon DAEMON               Daemon directory
      --client CLIENT               Client directory
      --config CONFIG               Config directory
      --logdir LOGDIR               Logging dir for daemon and client
      --scripts SCRIPTS             Scripts directory
      --template TEMPLATE           Template directory
      --dryrun                      Show what would have been done
      --create-uberjar              Force (re-)creating uberjar
      --start-on-system-boot        Install Windows genied.bat in startup folder
  -p, --port PORT             7888  Genie daemon port number (for start-on-system-boot)
  -v, --verbose                     Verbose output
  -h, --help                        Show help

With default locations:

item	default location	Related environment variables
java	<system>	GENIE_JAVA_CMD, JAVA_CMD, JAVA_HOME, java
daemon/jar	~/tools/genie, lein run	GENIE_DAEMON_DIR (also genied.sh)
config	~/.config/genie	GENIE_CONFIG_DIR (also templates)
log-dir	~/log	GENIE_LOG_DIR
templates	~/.config/genie/template	GENIE_TEMPLATE_DIR, GENIE_CONFIG_DIR
scripts	~/bin	GENIE_SCRIPTS_DIR
client	~/bin	GENIE_CLIENT_DIR

If you face issues creating an uberjar from the installer, try it directly with Leiningen:

cd genied
lein uberjar

The installer will try to overwrite binaries and scripts with new versions, but will not touch existing config files and templates.

Set environment variables

You might want to add the following environment vars to your .profile (see output of install.clj):

export GENIE_CLIENT_DIR=~/bin
export GENIE_DAEMON_DIR=~/tools/genie
export GENIE_JAVA_CMD=java
export GENIE_CONFIG_DIR=~/.config/genie
export GENIE_LOG_DIR=~/log
export GENIE_TEMPLATE_DIR=~/.config/genie/template
export GENIE_SCRIPTS_DIR=~/bin

Start automatically at system boot

Add a crontab entry so the Genie daemon starts automatically:

@reboot /home/your-user-name/tools/genie/genied.sh

Check genied.sh for giving java options like -Xmx.

Or, in Windows: see docs/windows.org.

On a Macbook, a process started with cron might not have all the rights the logged-in user has, e.g. with Onedrive. An alternative is to use the solution described in https://stackoverflow.com/questions/6442364/running-script-upon-login-mac:

• Paste the following one-line script into Script Editor: do shell script “$HOME/tools/genie/genied.sh”
• Then save it as an application.
• Finally add the application to your login items.

On first use the system will ask you for permissions to access e.g. Onedrive directory.

Test without installation

If you want to check out Genie without installing it, assuming you have Babashka and Leiningen installed (this uses ‘lein run’):

bb client/genie.clj --start-daemon
bb client/genie.clj test/test.clj -a

Testing new version

When you already have a version running, and possibly started at boot but want to try a new version:

genie --stop-daemon
cd genied
lein run

# in another terminal:
genie test/test.clj -a
test/run-all-tests.clj

Usage

An example script is shown below.

#! /usr/bin/env genie

(ns test
  (:require 
   [ndevreeze.cmdline :as cl]
   [clojure.data.csv :as csv]))

(def cli-options
  [["-c" "--config CONFIG" "Config file"]
   ["-h" "--help" "Show this help"]])

(defn data-csv
  [opt ctx]
  (println "Parsing csv using data.csv: " (csv/read-csv "abc,123,\"with,comma\"")))

(defn script [opt arguments ctx]
  (println "ctx: " ctx)
  (data-csv opt ctx))

;; expect context/ctx as first parameter, a map.
(defn main [ctx args]
  (cl/check-and-exec "" cli-options script args ctx))

;; for use with 'clj -m test-dyn-cl
(defn -main
  "Entry point from clj cmdline script"
  [& args]
  (cl/check-and-exec "" cli-options script args {:cwd "."})
  (System/exit 0))

A deps.edn should be in the same directory:

{:paths [""] ;; so script will be found in current dir, not in src-subdir.
 :deps
 {clojure.java-time/clojure.java-time {:mvn/version "0.3.2"}
  org.clojure/clojure {:mvn/version "1.10.1"}
  org.clojure/data.csv {:mvn/version "1.0.0"}}}

Then execute with Genie:

genie.clj ./test.clj

Or with clj:

clj -m test

Command line options

The genie.clj Babashka client has several options:

$ client/genie.clj -h
genie.clj - Babashka script to run scripts in Genie daemon
  -p, --port PORT                     7888  Genie daemon port number
  -m, --main MAIN                           main ns/fn to call. Empty: get from script ns-decl
  -l, --logdir LOGDIR                       Directory for client log. Empty: no logging
      --deps DEPS                           Use different deps.edn file
  -v, --verbose                             Verbose output
  -h, --help                                Show help
      --max-lines MAX-LINES           1024  Max #lines to read/pass in one message
      --noload                              Do not load libraries and scripts
      --nocheckdaemon                       Do not perform daemon checks on errors
      --nosetloader                         Do not set dynamic classloader
      --nomain                              Do not call main function after loading
      --nonormalize                         Do not normalize parameters to script (rel. paths)
      --list-sessions                       List currently open/running sessions/scripts
      --kill-sessions SESSIONS              csv list of (part of) sessions, or 'all'
      --start-daemon                        Start daemon running on port
      --stop-daemon                         Stop daemon running on port
      --restart-daemon                      Restart daemon running on port
      --max-wait-daemon MAX_WAIT_SEC  60    Max seconds to wait for daemon to start

Command line parameters

When we give command line parameters to a client script, these might be references to relative files. The client tries to convert these to absolute paths for the daemon:

• If it’s a dot (.) or starts with ./ it is converted to an absolute path
• If the parameter value exists as a local file, it is converted to an absolute path
• if –nonormalize is given, this conversion is not done.
• Scripts can use the (:cwd ctx) value to get the working directory of the script.

Creating a script

To create a script and deps.edn file from templates:

./scripts/genie_new.clj /path/to/new/script.clj

This uses template.clj and deps.edn from the template directory (GENIE_TEMPLATE_DIR). For more details see docs/background.org.

Testing

See directory ‘test’, with these scripts:

Test	Notes
run-all-tests.clj	Start a daemon, run all tests and stop daemon
bb_pipe.clj	Babashka test script for piping stdin->stdout
bb_stdout.clj	Babashka test script for generating delayed output
test_add_numbers.clj	Add numbers from cmdline
test.clj	Several tests with log, stdout, stderr
test_divide_by_0.clj	Test if exceptions are returned
test_dyn_cl.clj	Test dynamic class-loader
test_head.clj	Read a text file
test_load_file2.clj	Load/source a library, take 2
test_load_file.clj	Load/source a library, take 1
test_load_file_lib.clj	Library loaded by test_load_file(2).clj
test_log_concurrent.clj	Test if concurrent logs don’t get mixed up
test_loggers.clj	Test if loggers in script, client and daemon work
test_no_namespace.clj	Test without a script namespace
test_params.clj	Test command line parameters
test_stdin.clj	Test reading stdin
test_stdout_stderr.clj	Test output to stdout and stderr
test_two_namespaces.clj	Test with 2 namespaces in a file
test_write_file.clj	Test writing a text file

To run all these tests in the ‘test’ directory:

$ test/run-all-tests.clj -h
run-all-tests.clj - run all genie tests in this directory
  -p, --port PORT             7887  Genie daemon port number for test
  -l, --logdir LOGDIR               Directory for client log. Empty: no logging
  -v, --verbose                     Verbose output
  -h, --help                        Show help
      --clj                         Use clj instead of genie to run scripts
      --no-start-stop-daemon        Do not start a daemon before the tests

There is also a minimal Midje test for the daemon, calling test.clj as mentioned above:

cd genied
lein midje

Security

The daemon should run under a standard (non-root) user. All scripts are executed under this user’s credentials. The daemon only listens on localhost. In theory it should be possible to connect over the (local) network, but you probably do not want this. Also be aware Genie is not secure in a multi-user system: anyone can connect on the local port and the (local) netwerk traffic is not encrypted.

Todo

• See Issues on Github
• See docs/todo.org for future ideas.
• More real world using and testing

Related projects

Some Clojure-like languages having fast startup, but not all Clojure/JVM functionality:

• Babashka - Clojure implementation based on SCI.
• Closh - Shell comparable to Bash
• Fiji - from ImageJ, image processing, with Clojure scripting embedded.
• GraalVM - Compile to platform binaries
• Janet - own VM
• Joker - implementation in Go
• Hy - Python VM
• Lumo - JavaScript
• Pixie - own VM
• Planck - JavaScript

Earlier projects, some not actively maintained:

• Cake - merged with Leiningen
• Drip - Keeps a JVM in reserve.
• Grenchman - fast invocation of Clojure code over nREPL
• Inlein - mostly for setting up classpath, a new JVM is started for each script-run.
• Jark - seems offline. But Jark still exists.
• Lein-daemon - A leiningen plugin for daemonizing a clojure process (deprecated)
• Lein-jarbin - successor of lein-daemon
• Nailgun - client, protocol, and server for running Java programs from the command line without incurring the JVM startup overhead. See also the nice background information.
• QuickClojure - Python client, somewhat similar to Genie. Last update in 2015.
• Shevek - nRepl client made with Fennel (Lua).

And a discussion about some possibilities from 2016.

Daemon or agent

Maybe genied is more an agent then a daemon, according to e.g. https://www.aritsltd.com/blog/server/adding-startup-scripts-to-launch-daemon-on-mac-os-x-sierra-10-12-6/. A daemon runs as root, while an agent runs with the same rights as a user. Genied should run with user-rights, not as root.

More documentation

• docs/todo.org
• docs/background.org - If you want to know the details, and maybe want contribute
• docs/performance.org
• docs/windows.org - Specific issues when running on Windows.
• API Docs on Github pages

License

Copyright © 2021 Nico de Vreeze

Distributed under the Eclipse Public License, the same as Clojure.

See LICENSE