JS arbex implementation: parse, typecheck, compile expressions #4841

anandthakker · 2017-06-15T17:20:38Z

Todo:

anandthakker · 2017-06-17T02:51:55Z

@jfirebaugh @ansis plenty of work still left to do here, but a base implementation is in place in case you want to take an early look.

anandthakker · 2017-06-19T22:08:13Z

src/style-spec/function/expressions.js

+        name: 'object',
+        type: lambda(ObjectType, ValueType),
+        compile: fromContext('asObject')
+    },


json_array and object are assertions (yielding an error if the input isn't the right type), whereas number, string, boolean are coercions. Should we also have assertions for number/string/boolean?

jfirebaugh

Very cool stuff! I didn't thoroughly review everything, but wanted to submit what I noticed so far.

jfirebaugh · 2017-06-19T21:09:44Z

src/style-spec/function/expressions.js

+    };
+}
+
+function defineBinaryMathOp(name, isAssociative) {


None of the current uses of defineBinaryMathOp pass a second argument.

Ah, whoops! This was intended for allowing +, &&, etc. to take > 2 args as a convenience.

Done in 2b91368

jfirebaugh · 2017-06-19T21:11:50Z

src/style-spec/function/expressions.js

+    };
+}
+
+function defineComparisonOp(name, isAssociative) {


Ditto defineComparisonOp.

jfirebaugh · 2017-06-19T21:18:57Z

src/style-spec/function/expressions.js

+ }
+ */
+
+const expressions: { [string]: Definition } = {


[ExpressionName]: Definition

Ack. Seems like this isn't compatible with accessing this map using values pulled from a mixed array. There's no good way to take myRawJSONExpression[0] and refine it so that flow knows that it's an ExpressionName.

jfirebaugh · 2017-06-19T21:41:46Z

src/style-spec/function/evaluation_context.js

+    },
+
+    typeOf: function (x) {
+        if (Array.isArray(x)) return 'Vector<Value>';


Is this the only place generics syntax is exposed publicly? I wonder if Array is a better public name for this type, for familiarity. Internally we could rename Vector<T> ⇢ Array<T> and Array<T, N> ⇢ FixedArray<T, N>.

It's not quite the only place it's exposed publicly: type checking error messages will need to refer to types by name, and probably some runtime error messages will, too. That said, I'm also with you in that I've been uneasy/unsatisfied with having Vector<Value> as a possible output of typeof (because of both the generics syntax and the Value alias).

Would it be confusing for [typeof, [get, 'foo']] to equal Array, but then to have [number, [at, [get, 'foo'], 0]] to produce a runtime error message like Expected Vector<number> but properties.foo was an object instead?

Maybe we could always print names like:

Value: JSONValue

Vector: T[]

Array<T, N>: T[N]

Or, if we think T[] is too concise and easy to misread, we could do:

Vector: Array of T's

Array<T, N>: Array of N T's

jfirebaugh · 2017-06-19T22:37:31Z

src/style-spec/function/expressions.js

+            const result = [];
+            while (args.length > 1) {
+                const c = args.splice(0, 2);
+                result.push(`${c[0].js} ? ${c[1].js}`);


Parentheses are needed around subexpressions, either in individual compile definitions or inserted globally by the compiler. Else you'll get the wrong precedence when compiling things like ["&&", a, ["case", b, c, d]] -- desired precedence is a && (b ? c : d), but without parentheses a && b ? c : d parses as (a && b) ? c : d.

👍 yep -- it's done globally at the end of compile(), 04dbb97#diff-442324b3174c25e635f18536d7732df2R147, but this reminds me, we should include a test that targets this since it's be very easy to regress

Done in 013a59e

jfirebaugh · 2017-06-19T23:36:57Z

src/style-spec/function/compile.js

+ *
+ * @private
+ */
+function compileExpression(expr: any) {


Looks like it's possible to narrow any to mixed here without too much trouble.

jfirebaugh · 2017-06-19T23:37:18Z

src/style-spec/function/compile.js

+     isFeatureConstant: boolean,
+     isZoomConstant: boolean,
+     expression: TypedExpression,
+     function?: any


This any can be Function.

jfirebaugh · 2017-06-20T18:07:28Z

For consistency with get/has, we should also reorder at's arguments

anandthakker · 2017-06-21T02:40:14Z

docs/style-spec/expressions.md

@@ -134,7 +134,7 @@ Every expression evaluates to a value of one of the following types.
 ###Decision:
 - `["case", cond1: Boolean, result1: T, cond2: Boolean, result2: T, ..., cond_m, result_m: T, result_otherwise: T] -> T`
 - `["match", x: T, a_1: T, y_1: U, a_2: T, y_2: U, ..., a_m: T, y_m: U, y_else: U]` - `a_1`, `a_2`, ... must be _literal_ values of type `T`.
- `["is_error", expr: T]` - `true` if `expr` is an `Error` value, `false` otherwise
+- `["coalesce", e1: T, e2: T, e3: T, ...] -> T` - evaluates each expression in turn until the first non-error value is obtained, and returns that value.


This should probably be "non-error and non-null"

anandthakker · 2017-06-22T20:22:06Z

If a user specifies a property function with, say, numeric stop outputs, for a style property whose spec definition specifies a string, we accept the mistyped function and just yield the default. IMO, this should be a validation error instead.

jfirebaugh · 2017-06-27T03:30:24Z

When I look at real world examples like in #4836, I want to be able to extract array and object literals as constants, assigned in "let" expressions, and then write the guts of the expression with a data-driven dispatch into the constant. So, yes, if we make them possible/convenient, I think there will be use cases for arrays and objects outside of (a) and (b).

If/when we do support this for both objects and arrays, they should be parallel in syntax. Here are some possibilities:

["array", [...]] and ["object", {...}], with no evaluation "inside" the argument.
["array", value, value...], and ["object", string, value, string, value...], with evaluation of arguments.
["quote", [...]] and ["quote", {...}], with no evaluation "inside" the argument. A lispish single special form; provides the option to have "quasiquote" form in the future for partial inner evaluation. I also like this form because it leaves the names "array" and "object" free for assertion/coercion. We could choose less lispish names if you prefer, for example "quote" ⇢ "literal", "quasiquote" ⇢ "template", "unquote" ⇢ "substitute".

jfirebaugh · 2017-06-27T03:41:32Z

For the primitive types, I think coercion will be used much more commonly than assertion, which makes me want coercion to be as concise as possible

Funny, I have the opposite instinct. I would think assertion/annotation to be the more common (especially if we expect Studio to auto-insert annotations based on tilestats), and coercion to be the more dangerous, and hence better suited to the less concise name.

jfirebaugh · 2017-06-27T04:34:33Z

src/style-spec/function/expressions.js

+    'string': {
+        name: 'string',
+        type: lambda(StringType, ValueType),
+        compile: args => ({ js: `String(${args[0].js})` })


Is it wise to inherit all of JS's coercion quirks?

["string", ["^", 2, 70]] ⇢ "1.1805916207174113e+21"

["string", ["properties"]] ⇢ "[object Object]"

["string", ["array", 1, 2]] ⇢ "1,2"

["string, ["get", "weird"]] + {"properties": {"weird": {"toString": "asdf"}} ⇢ TypeError

I do think it seems reasonable to lean on the platform's number->string conversion... as for objects and arrays... would it be reasonable to just punt, requiring primitive types only?

anandthakker · 2017-06-27T11:58:56Z

Funny, I have the opposite instinct. I would think assertion/annotation to be the more common (especially if we expect Studio to auto-insert annotations based on tilestats)

Good point -- I was thinking more that coercion would be more common in manually-authored expressions, which is why I was going for conciseness...

coercion to be the more dangerous, and hence better suited to the less concise name.

... but this is convincing. 👍 for making coercion less concise / more explicit.

1ec5 · 2017-06-27T19:15:21Z

Good point -- I was thinking more that coercion would be more common in manually-authored expressions, which is why I was going for conciseness...

This was my thought as well. Runtime styling doesn’t have the benefit of auto-annotation via tilestats. Per mapbox/mapbox-gl-native#8074 (comment), I don’t think developers on iOS will mind a little danger if they can avoid mandatory type annotations (which would have major usability issues).

anandthakker · 2017-06-30T16:31:32Z

src/style-spec/function/evaluation_context.js

-            }
-            type = (!itemtype || itemtype === 'Value') ?
-                'Array' : `Array<${titlecase(itemtype)}>`;
+            const t = ArrayLiteral.inferArrayType(items);


@jfirebaugh This change causes the runtime typing of [get, foo] when foo is an array to be as specific as possible, based on its actual contents:

the array is homogenous of type number, string, or boolean, then => Array<typeof item, properties.foo.length>

otherwise => Array<Value, properties.foo.length>

anandthakker · 2017-07-01T00:27:15Z

src/style-spec/function/expression.js


-        return array(itemType || ValueType, value.length);
-    }
+        if (!isValue(args[0]))


As of now, there's actually no way to express a color literal -- rgba and color are functions -- but this would allow ['literal', new Color(...)].

Yeah, I figured that would be okay -- since we're not likely to export Color publicly, there no way to construct such an expression. And even if there were, it'd be harmless to do so.

anandthakker · 2017-07-01T00:28:19Z

src/style-spec/function/expression.js

@@ -145,12 +145,12 @@ function parseExpression(expr: mixed, context: ParsingContext) : Expression {
        return new LiteralExpression(key, NumberType, expr);
    } else if (Array.isArray(expr)) {
        if (expr.length === 0) {
-            throw new ParsingError(key, `Expected an array with at least one element.`);
+            throw new ParsingError(key, `Expected an array with at least one element. If you wanted a literal array, use ["literal", []].`);


👍 to this and the others like it.

jfirebaugh · 2017-07-01T01:13:55Z

Now that we have an AST made up of Expression instances, we should define things like isFeatureConstant and isZoomConstant as independent recursive functions rather than lumping them in with the compilation step.

Expression#compile itself should be typed () => string | Array<CompileError>.

anandthakker · 2017-07-02T01:43:37Z

Now that we have an AST made up of Expression instances, we should define things like isFeatureConstant and isZoomConstant as independent recursive functions rather than lumping them in with the compilation step.
Expression#compile itself should be typed () => string | Array.

💯 I'm doing this as part of implementation of let.

anandthakker · 2017-07-03T01:19:24Z

Few notes/questions about let:

Uses let (as opposed to let* or letrec) semantics.
We could make Reference part of LiteralExpression and LetExpression a subclass of LambdaExpression. I kept the former separate for semantic reasons, and the latter also because it requires specialized type checking.
To keep things simple for now, a [zoom] curve is not allowed in a let binding; it can only appear in the result of a let, even though we theoretically could accept, e.g. [let, 'a', [curve, [linear], [zoom], 0, 0, 10, 10], ['a']]
Currently, a let binding that produces an error will cause the entire let expression to be an error, even if it would have been unused. E.g. [let, 'a', [number, [get, x]], [case, false, [a], 0]]. What's more natural: the present behavior, or only having an error propagated if the erroneous binding is actually referenced?

anandthakker · 2017-07-03T16:20:47Z

src/style-spec/function/expression.js

+    constructor(key: string, name: string, type: Type) {
+        super(key, type);
+        if (!/^[a-zA-Z_]+[a-zA-Z_0-9]*$/.test(name))
+            throw new ParsingError(key, `Invalid identifier ${name}.`);


This check should be moved to LetExpression.parse(), and it should also either check against JS reserved words or else 'escape' them (e.g. replace function with var_function or something)

anandthakker · 2017-07-03T21:20:06Z

Squashed this down, FF merging into feature/expressions branch

anandthakker force-pushed the compile-expressions branch 2 times, most recently from e72044a to ea2521b Compare June 15, 2017 20:14

anandthakker mentioned this pull request Jun 15, 2017

Project Arbex (Arbitrary Expressions) #4777

Merged

15 tasks

anandthakker force-pushed the compile-expressions branch 5 times, most recently from 72237fa to 38f0058 Compare June 17, 2017 02:47

anandthakker force-pushed the compile-expressions branch 4 times, most recently from 637f4af to 0a272d4 Compare June 19, 2017 20:37

anandthakker commented Jun 19, 2017

View reviewed changes

jfirebaugh reviewed Jun 19, 2017

View reviewed changes

anandthakker force-pushed the compile-expressions branch 4 times, most recently from 2d49f8e to 69ca97a Compare June 20, 2017 12:47

anandthakker force-pushed the compile-expressions branch from c98700d to 834a6c6 Compare June 21, 2017 02:38

anandthakker commented Jun 21, 2017

View reviewed changes

anandthakker force-pushed the compile-expressions branch 4 times, most recently from 8d1e797 to d2692a0 Compare June 22, 2017 15:49

anandthakker force-pushed the compile-expressions branch from dd09152 to 10cec45 Compare June 22, 2017 20:42

anandthakker changed the title ~~WIP - JS arbex implementation: parse, typecheck, compile expressions~~ JS arbex implementation: parse, typecheck, compile expressions Jun 22, 2017

anandthakker force-pushed the compile-expressions branch from 10cec45 to ca9d55d Compare June 22, 2017 21:25

jfirebaugh reviewed Jun 27, 2017

View reviewed changes

1ec5 mentioned this pull request Jun 27, 2017

Reimplement MGLStyleValue atop NSExpression mapbox/mapbox-gl-native#8074

Closed

anandthakker force-pushed the feature/expressions branch from c289e5f to b75efe5 Compare June 28, 2017 12:01

anandthakker force-pushed the compile-expressions branch 2 times, most recently from 074b108 to 8f9f2df Compare June 30, 2017 01:38

anandthakker force-pushed the feature/expressions branch from b75efe5 to c1c29cb Compare June 30, 2017 12:27

anandthakker force-pushed the compile-expressions branch 2 times, most recently from cc10399 to ab4c1fb Compare June 30, 2017 16:25

anandthakker commented Jun 30, 2017

View reviewed changes

anandthakker commented Jul 1, 2017

View reviewed changes

anandthakker force-pushed the compile-expressions branch from 0fd95a5 to f1b8ceb Compare July 3, 2017 01:02

anandthakker force-pushed the compile-expressions branch from f1b8ceb to 10eb188 Compare July 3, 2017 01:21

anandthakker commented Jul 3, 2017

View reviewed changes

anandthakker force-pushed the compile-expressions branch from b0203b0 to 19c47dc Compare July 3, 2017 16:55

Anand Thakker added 4 commits July 3, 2017 17:17

Parse, typecheck, and compile expressions

1c8b68f

Set up integration test system for expressions

8b4a80f

Update expressions.md

efa6f29

Accept {expression: [...]} in style spec validation

6ba399f

anandthakker force-pushed the compile-expressions branch from 3a2fab2 to 6ba399f Compare July 3, 2017 21:19

anandthakker merged commit 6ba399f into feature/expressions Jul 3, 2017

jfirebaugh deleted the compile-expressions branch September 21, 2017 17:02

JS arbex implementation: parse, typecheck, compile expressions #4841

JS arbex implementation: parse, typecheck, compile expressions #4841

Conversation

anandthakker commented Jun 15, 2017 • edited Loading

anandthakker commented Jun 17, 2017

Choose a reason for hiding this comment

jfirebaugh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anandthakker Jun 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfirebaugh commented Jun 20, 2017

anandthakker Jun 21, 2017 • edited Loading

Choose a reason for hiding this comment

anandthakker commented Jun 22, 2017

jfirebaugh commented Jun 27, 2017

jfirebaugh commented Jun 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anandthakker commented Jun 27, 2017

1ec5 commented Jun 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfirebaugh commented Jul 1, 2017

anandthakker commented Jul 2, 2017

anandthakker commented Jul 3, 2017

Choose a reason for hiding this comment

anandthakker commented Jul 3, 2017

anandthakker commented Jun 15, 2017 •

edited

Loading

anandthakker Jun 20, 2017 •

edited

Loading

anandthakker Jun 21, 2017 •

edited

Loading