HISTORY.md

Morel release history and change log

For a full list of releases, see github.

0.4 / 2024-01-04

Release 0.4 extends from syntax, adding suchthat, take and skip clauses; allows identifiers to be quoted using backticks; improves pretty-printing of tuples and long lines.

Contributors: Julian Hyde, Rette66

Features

• [MOREL-204] Add take and skip relational operators
• [MOREL-209] File reader, and progressive types
• [MOREL-210] Fold long types when printing
• Rename Option.flatten to Option.join
• [MOREL-71] Allow identifiers to be quoted using backticks
• [MOREL-206] Indent tuples when printing
• [MOREL-129] Define relations via constrained iterations, and introduce a suchthat keyword to use them

Bug-fixes and internal improvements

• Add directory property to Session
• Add type parameter to Core.Literal.unwrap() method
• [MOREL-208] FromBuilder should remove trivial yield step between two scan steps
• [MOREL-205] Pattern that uses nested type-constructors should not be considered redundant
• Add utility methods Static.transform and transformEager
• [MOREL-203] Cannot deduce type for from b in [SOME true, NONE]
• Refactor: In Unifier.Variable, store ordinal rather than computing it
• Refactor: Rename Ord.forEach to forEachIndexed
• In CoreBuilder, add tuple(TypeSystem, Exp...), a more convenient apply
• Simplify EnvShuttle, EnvVisitor by adding push method
• Add interface PairList and interface ImmutablePairList
• Add method Static.nextPowerOfTwo
• [MOREL-201] Real.signBit gives different result on JDK 19/ARM

Build and tests

• Run script.sml in tests
• [MOREL-207] Detect and fix flaky tests
• Disallow static star import
• [MOREL-200] In the test suite, run arbitrary "lint" checks on code
• [MOREL-198] Idempotent mode for test scripts
• Add a test for various closure and recursion scenarios
• Allow CI runs to be triggered manually, and monthly

Component upgrades

• Bump build-helper-maven-plugin from 3.3.0 to 3.5.0
• Bump calcite from 1.32.0 to 1.36.0
• Bump checkstyle from 10.3.4 to 10.12.7
• Bump guava from 31.1-jre to 33.0.0-jre; increase minimum guava version from 19.0 to 21.0
• Bump hsqldb from 2.7.0 to 2.7.2
• Bump javacc from 7.0.12 to 7.0.13
• Bump jdk to 21; minimum jdk is still 8
• Bump jline from 3.21.0 to 3.25.0
• Bump junit-jupiter from 5.9.1 to 5.10.1
• Bump maven-checkstyle-plugin from 3.2.0 to 3.3.1
• Bump maven-compiler-plugin from 3.10.1 to 3.12.1
• Bump maven-enforcer-plugin from 3.1.0 to 3.4.1
• Bump maven-javadoc-plugin from 3.4.1 to 3.6.3
• Bump maven-project-info-reports-plugin from 3.4.1 to 3.5.0
• Bump maven-release-plugin from 2.4.2 to 3.0.1
• Bump maven-scm-provider-gitexe from 1.9.1 to 2.0.1
• Bump maven-site-plugin from 3.12.1 to 4.0.0-M13
• Bump maven-source-plugin from 3.2.1 to 3.3.0
• Bump maven-surefire-plugin from 2.22.2 to 3.2.3
• Bump slfj from 2.0.3 to 2.1.0-alpha1
• Add versions-maven-plugin version 2.16.2
• [MOREL-199] Support JDK 19 and 20

Site and documentation

• [MOREL-211] Release 0.4
• Add Maven Central badge to README.md

0.3 / 2022-10-02

Release 0.3 extends from syntax, adding an order clause, allowing multiple yield, group and where steps in any order, variable = value, patterns in in. The compiler now detects redundant and exhaustive matches, and supports views and inlining. We add the use function and standard structures Math and Real to the standard library. Integration with Apache Calcite allows us to translate whole programs to relational algebra.

Contributors: Gabriel Tejeda, Gavin Ray, Julian Hyde, Sergey Nuyanzin

Features

• [MOREL-165] Improve message when type resolution cannot deduce full record type
• [MOREL-159] from should not have a singleton record type unless it ends with a singleton record yield
• [MOREL-147] Allow Calcite schemas to contain sub-schemas
• [MOREL-138] Type annotations in patterns, function declarations and expressions
• [MOREL-55] Analyze match coverage, detecting redundant and exhaustive matches
• Satisfiability prover
• [MOREL-118] Report positions in error messages and exceptions
• [MOREL-105] Allow identifiers to contain prime characters (')
• [MOREL-70] Polymorphic datatype
• [MOREL-104] Make functions top-level
• [MOREL-103] Layered patterns (as), and composite val
• [MOREL-102] Real structure
• Allow quick eager evaluation for functions/operators with more than one argument
• [MOREL-96] Print control
• [MOREL-100] Allow double-quote and backslash in string and char literals
• [MOREL-88] Math structure
• Floating point: posInf, negInf, NaN, and negative zero
• [MOREL-94] Wordle solver
• [MOREL-95] Mutually recursive functions
• Add Relational.iterate, which allows "recursive queries" such as transitive closure
• [MOREL-69] Add compute clause, for monoid comprehensions
• [MOREL-86] Add use function, to read and execute source from a file
• [MOREL-72] Join
• [MOREL-11] In from clause, allow 'variable = value'
• [MOREL-62] Add function Relational.only, which allows scalar sub-queries
• [MOREL-65] In the shell (REPL), use prompt '-' and continuation '='
• [MOREL-51] Add Relational.exists function, and push it down to Calcite
• Add morel.lang, Morel language definition for the Linux 'highlight' command
• [MOREL-60] Push elem, notelem and not ... elem down to Calcite (as SQL IN and NOT IN)
• [MOREL-52] Allow multiple yield steps in from
• [MOREL-45] Translate List.filter as if user had written where; and List.map to yield
• [MOREL-57] group with empty key should emit one row
• [MOREL-54] Views (parameterized functions that return a relation) and inlining
• [MOREL-53] Optimize core language by inlining expressions
• [MOREL-42] Add a morel Calcite UDF, to allow hybrid plans with a mixture of relational and non-relational
• [MOREL-40] Translate to full Calcite algebra
• [MOREL-46] Core language
• [MOREL-48] Deduce types when a function is applied to a record selector
• [MOREL-44] Session variables
• [MOREL-41] Add Sys.plan () function, to display plans from the shell
• [MOREL-39] Implement built-in vector and order data types
• [MOREL-38] Implement built-in option datatype and supporting functions
• [MOREL-37] Raise exceptions in built-in functions
• [MOREL-25] Add o (function composition) and @ (list concatenation) operators
• [MOREL-36] In record pattern, make labels optional, and disallow ... anywhere but end
• [MOREL-34] Functions in relations
• [MOREL-27] Create objects for built-in structures List and String, and allow structure.name references
• [MOREL-33] Add elem, notelem operators
• [MOREL-30] Add union, intersect, except operators
• [MOREL-31] In from expression, allow in to assign to pattern
• [MOREL-29] Overload + operator to allow both int and real arguments
• [MOREL-24] Use = rather than as for assigning column aliases in group and compute
• [MOREL-23] Default arguments for aggregate functions
• [MOREL-17] Allow from clause that defines 0 sources
• [MOREL-20] Add order clause
• In compute clause of group, apply of expression to each row
• [MOREL-21] In from, allow multiple group and where clauses in any order, optionally followed by yield

Bug-fixes and internal improvements

• [MOREL-179] In compilation environment, use name + ordinal as the key, not just name, to accommodate variable copies caused by inlining
• Refactor: Move list methods into Static
• In CoreBuilder, add methods for frequently used operations: equals, lessThan, only etc.
• Allocate bool literals (true and false) once in CoreBuilder
• Keywords should be lower case
• [MOREL-101] Join variables are out of order in the rows sent to an aggregate function
• [MOREL-99] Script parser hangs if it encounters invalid syntax, and ignores comments at ends of files
• Refactor: Array to List, List to Consumer
• Refactor: move Shell state into new interface Config
• Tuning
• Rename Option.join to flatten (to make way for the join keyword)
• [MOREL-73] Keywords must be lower case
• [MOREL-64] order after yield gives 'unbound variable' error
• [MOREL-67] In compute clause, the key value should be available but is null
• During inlining, use '_' as delimiter for qualified variable names
• [MOREL-59] Converting an empty list to relational algebra throws
• Add Core.Yield step, and obsolete Core.yieldExp
• Rationalize order of Java 'import' statements
• Always inline atomic variables
• Add method Static.shorterThan(Iterable, int)
• [MOREL-49] Function argument () should have type unit, not 0-tuple
• [MOREL-50] Type resolver does not resolve all fields of record types
• Improve field names, e.g. rename Exp e to Exp exp
• Remove overrides of toString() method in Environment and EvalEnv
• Factor Converter out of CalciteForeignValue
• Refactor: rename let fields
• Add enum Prop, for strongly-typed properties
• [MOREL-13] Garbage-collect obscured variables
• AST that contains a call to op + is unparsed incorrectly
• Refactor DataSet
• TypeVisitor should visit parameter types inside DataType
• [MOREL-35] Record pattern in from mixes up fields if not in alphabetical order
• Deduce type of polymorphic field in tuple or record
• Add tracing to Unifier
• Move class TypeResolver.TypeMap to top-level
• Correct order of numeric labels in records, and allow 0 as a label
• A record with a field named "1" is a record, not a tuple
• During validation, replace From.sources if rewrites occur
• [MOREL-28] The sum aggregate function only works on int values
• [MOREL-26] Cannot parse () as pattern
• Morel shell should work even if Apache Maven is not installed
• [MOREL-19] AssertionError: unknown FUN_DECL
• [MOREL-22] Character literal should unparse as #"a", not "a"

Build and tests

• Bump slf4j-api from 2.0.2 to 2.0.3
• Bump hsqldb from 2.5.1 to 2.7.0
• Add hsqldb-version parameter to CI
• Add interface Tracer, so that tests can check for several events during the lifecycle
• Bump git-commit-id-plugin from 2.1.9 to 4.9.10
• Bump checkstyle from 7.8.2 to 10.3.4, maven-checkstyle-plugin from 3.0.0 to 3.1.2, maven-source-plugin from 2.2.1 to 3.2.1, maven-compiler-plugin from 2.3.2 to 3.10.1, build-helper-maven-plugin from 1.9 to 3.3.0
• Bump calcite-core from 1.29.0 to 1.32.0
• Make ShellTest more robust
• Turn off Travis CI
• Bump hsqldb from 2.3.1 to 2.5.1, foodmart-data-hsqldb from 0.4 to 0.5, scott-data-hsqldb from 0.1 to 0.2
• Bump maven-project-info-reports-plugin from 2.9 to 3.4.1
• Bump maven-enforcer-plugin from 3.0.0-M1 to 3.1.0
• Bump maven-javadoc-plugin from 3.0.1 to 3.4.1
• Bump javacc-maven-plugin from 3.0.0 to 3.0.3
• Test Guava versions 19.0 to 31.1-jre in CI
• Bump guava from 21.0 to 23.0
• Bump jline from 3.16.0 to 3.21.0
• Bump maven-surefire-plugin from 3.0.0-M3 to 3.0.0-M7
• Bump jsr305 from 1.3.9 to 3.0.2
• Bump slf4j-api from 1.7.25 to 2.0.2
• Bump junit-jupiter.version from 5.7.2 to 5.9.1
• Bump javacc from 7.0.5 to 7.0.12
• Enable Dependabot
• Run CI on multiple Java versions, with Javadoc
• [MOREL-107] ShellTest is non-deterministic
• Add GitHub workflow to build and test
• Move project to 'hydromatic' GitHub organization
• Bump maven from 3.6.3 to 3.8.4
• Upgrade calcite-core to 1.29
• In parser tests, make sure there are no characters after the expression being parsed
• Refactor: Test fixture for ShellTest
• Refactor: Move ScriptTest.Utils to top-level class TestUtils
• Bump jline from 3.12.1 to 3.16.0, to give support for xrvt terminals
• Add tests for 'StrangeLoop 2021' talk
• Travis: quote variables, skip install
• Split hybrid.sml out of foreign.sml
• Docker login
• LearningMatcher makes it easier to ensure that two queries have the same plan or same results
• In tests, move implementations of interface Matcher into new utility class Matchers
• Bump calcite-core to 1.27
• Run tests in parallel
• [MOREL-47] Bump junit from 4.11 to 5.7.2, hamcrest from 1.3 to 2.2
• Add tests for blog post 'Aggregate queries in Morel'
• Add tests for blog post 'Word Count revisited'
• Bump javacc from 4.0 to 7.0.5, maven-javacc-plugin from 2.4 to 3.0.0

Site and documentation

• [MOREL-181] Release 0.3
• Update operator list in README
• In reference, re-order the String and Vector built-in functions
• Add example of matching regular expressions using combinators
• Add missed brackets and semicolons in README
• Improve Morel picture on home page
• Add functions and types to Morel language reference
• Add build status to home page
• Another example of a recursive query: Floyd-Warshall algorithm for shortest path in a weighted graph
• Add examples of recursive queries and fixed-point algorithms
• Add Morel logo and square image

0.2 / 2020-03-10

The first release since smlj was renamed to Morel includes major improvements to the type system and relational extensions. Some highlights:

• Functions and values can have polymorphic types, inferred as part of a Hindley-Milner type system;
• Relational expressions may now include a group clause, so you can evaluate aggregate queries (similar to SQL GROUP BY);
• Foreign values allow external data, such as the contents of a JDBC database, to be handled as if it is in memory;
• Add built-in functions based on the String and List structures in the Standard ML basis library;
• Postfix field reference syntax makes Morel more familiar to SQL users;
• Add Morel language reference.

Features

• [MOREL-15] Improve pretty-printing: wrap long lines, and abbreviate long lists and deep structures
• [MOREL-12] In from clause, allow initializers to reference previous variables
• In group, as alias is optional
• [MOREL-10] Implicit labels in record expressions
• [MOREL-9] Allow <expr>.<field> as an alternative syntax for #<field> <expr>
• [MOREL-7] Rename project from 'smlj' to 'morel'
• [SMLJ-5] Foreign values, including record values based on the contents of a JDBC schema
• [SMLJ-6] Add group clause (and compute sub-clause) to from expression, to support aggregation and aggregate functions
• Polymorphic types
• Add String and List basis functions

Bug-fixes and internal improvements

• [MOREL-16] Ensure that types derived for REPL expressions have no free type variables
• [MOREL-14] Tuple should equal record, and both equal unit when empty
• Add macros (special built-in functions that are 'called' at compile time to generate a new AST)
• Add interface MutableEvalEnv, for code that wants to mutate the last binding in an environment
• Make EvalEnv immutable
• Recursive functions in closures use the wrong environment
• Unit literal's toString() should be (), not []
• For built-ins, add their alias to the compile-time environment
• In parallel declarations (let ... and) add variables to compilation environment
• Refactor special type constructors (list, tuple, record)
• [SMLJ-4] let fun inside from expression fails
• Move built-in constants and functions into new enum BuiltIn
• In Shell, fix parsing single-quote in line comments, and line endings in multi-line statements

Build and tests

• Test expressions used in documentation and blog posts
• Example of a user-defined aggregate function in a query
• Add a test with a temporary function in a query that takes a record-valued argument
• In ScriptTest, only load Dictionary if script is foreign.sml
• Enable some group tests
• Add Sys_env () function, that returns the current environment
• Upgrade maven: 3.5.4 → 3.6.3
• Add maven wrapper jar
• Use correct maven-javadoc-plugin version
• Before launching repl, build test as well as main
• Convert MainTest to use fluent style
• In Shell, add optional pause, which seems to make ShellTest deterministic

Site and documentation

• Add Morel language reference
• Add image to README
• Add javadoc to site
• Generate an asciinema demo

0.1 / 2019-07-24

Initial release features the core language (primitive types, lists, tuples, records; let, if, fn and case expressions; val, fun and datatype declarations), an interactive shell smlj, and relational extensions (from).