Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate AstRewriterBase to PIG's VisitorTransform #356

Merged
merged 15 commits into from
Jan 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions cli/src/org/partiql/cli/Repl.kt
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@ import com.amazon.ion.system.*
import com.amazon.ionelement.api.toIonValue
import org.partiql.cli.ReplState.*
import org.partiql.lang.*
import org.partiql.lang.ast.*
import org.partiql.lang.ast.passes.MetaStrippingRewriter
import org.partiql.lang.eval.*
import org.partiql.lang.syntax.*
import org.partiql.lang.util.*
Expand Down
4 changes: 2 additions & 2 deletions docs/dev/README-AST-INTRO.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,11 @@ result of the `when` becomes the function's return value:
```Kotlin
fun transformNode(exprNode: ExprNode): ExprNode = when(exprNode) {
is Literal -> { // case() is not needed
//Perform cloning rewrite
//Perform cloning transform
...
}
is VariableReference -> { // case() is not needed
//Perform cloning rewrite
//Perform cloning transform
...
}
//and so on for all nodes
Expand Down
2 changes: 1 addition & 1 deletion examples/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ List of Examples:
* EvaluationWithLazyBindings: query evaluation with global bindings that are lazily evaluated
* ParserErrorExample: inspecting errors thrown by the `Parser`
* ParserExample: how to parse a query and serialize the query AST
* PartialEvaluationRewriter: simple rewriter example that partially evaluates simple operations like `1+1`
* PartialEvaluationVisitorTransform: simple visitor transform example that partially evaluates simple operations like `1+1`
* PreventJoinVisitor: visitor example to validate a query, in this case it prevents queries with `JOIN`
* SimpleExpressionEvaluation: how to run a simple query
* Java:
Expand Down
2 changes: 1 addition & 1 deletion examples/src/kotlin/org/partiql/examples/ParserExample.kt
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ class ParserExample(out: PrintStream) : Example(out) {
// Use the SqlParser instance to parse the example query. This is very simple.
val originalAst = parser.parseExprNode(query)

// Now the originalAst can be inspected, rewritten and/or serialized as needed.
// Now the originalAst can be inspected, transformed and/or serialized as needed.
// For now we're just going to serialize and deserialize it.

// Convert the ExprNode AST to the Ion s-expression form.
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
package org.partiql.examples

import com.amazon.ion.IonSystem
import com.amazon.ion.system.IonSystemBuilder
import com.amazon.ionelement.api.toIonElement
import org.partiql.examples.util.Example
import org.partiql.lang.CompilerPipeline
import org.partiql.lang.ast.toAstStatement
import org.partiql.lang.ast.toExprNode
import org.partiql.lang.domains.PartiqlAst
import org.partiql.lang.eval.CompileOptions
import org.partiql.lang.eval.EvaluationSession
import org.partiql.lang.syntax.SqlParser
import java.io.PrintStream



/**
* A simple AST visitor transform that performs partial evaluation--i.e.: evaluates all sub-expressions containing only
* literal operands and replaces them with the result. For example, the query `1 + 2 * 3` would be transformed to
* simply `7`. (That is, `(+ 1 (* (lit 2) (lit 3)))` would be transformed to `(lit 7)`) (Note: s-expression AST is
* shown without wrapping `term` or `meta`.)
*
* The query `foo + 2 * 3` would be transformed to `foo + 6` (`(+ (id foo case_insensitive) (* 2 3)))`
* becomes `(+ (id foo case_insensitive) (lit 6))`
*
* This example just shows the partial evaluation for addition. Once these operations are better modeled in the
* PIG domain (https://github.com/partiql/partiql-lang-kotlin/issues/241), this can be expanded to all NAry operations
* easily.
*
* This isn't suitable for production use primarily because it needs significantly more testing and
* it doesn't go as far as it could:
* - function calls with only literal arguments could also be transformed, assuming the function is deterministic
* - cast operations against literal values, e.g.: `CAST(1 AS string)` are not considered literals, even though they
* could.
* - the transform will only be applied if all of the arguments of an NAry AST node are literals, however
* a production quality version of this should be able to transform this:
*
* ```
* (+
* (lit 1)
* (lit 2)
* (id foo case_insensitive)
* (lit 3)
* (lit 4))
* ```
*
* Into:
*
* ```
* (+
* (lit 3)
* (id foo case_insensitive)
* (lit 7))
* ```
*
* @param ion should be the same instance of [IonSystem] that was passed to [SqlParser].
* @param compileOptions should be the same instance of [CompileOptions] that was/will be passed to
* [EvaluatingCompiler].
*/
private class PartialEvaluationVisitorTransform(val ion: IonSystem, val compileOptions: CompileOptions) : PartiqlAst.VisitorTransform() {
private val pipeline = CompilerPipeline.build(ion) { compileOptions(compileOptions) }
private val session = EvaluationSession.standard()

override fun transformExprPlus(node: PartiqlAst.Expr.Plus): PartiqlAst.Expr {
val ops = node.operands
val metas = node.metas

val transformedOps = ops.map { transformExpr(it) }
val transformedNAry = PartiqlAst.build { plus(transformedOps, metas) }

return when {
transformedOps.all { it is PartiqlAst.Expr.Lit } -> {
val e = pipeline.compile(PartiqlAst.build { query(transformedNAry) }.toExprNode(ion) )
val partiallyEvaluatedResult = e.eval(session)
PartiqlAst.build { lit(partiallyEvaluatedResult.ionValue.toIonElement(), metas) }
}
else -> {
transformedNAry
}
}
}
}

/** Demonstrates how to use [PartialEvaluationVisitorTransform] as part of a [CompilerPipeline]. */
class PartialEvaluationVisitorTransformExample(out: PrintStream) : Example(out) {
private val ion = IonSystemBuilder.standard().build()

override fun run() {
val pipeline = CompilerPipeline.build(ion) {
addPreprocessingStep { node, stepContext ->
val visitorTransformer = PartialEvaluationVisitorTransform(ion, stepContext.compileOptions)

val originalAst = node.toAstStatement()
print("Original AST:", originalAst.toString())

val query = node.toAstStatement() as PartiqlAst.Statement.Query

val transformedNode = PartiqlAst.build { query(visitorTransformer.transformExpr(query.expr)) }
print("Transformed AST:", transformedNode.toString())

transformedNode.toExprNode(ion)
}
}

pipeline.compile("1 + 1")
}
}
2 changes: 1 addition & 1 deletion examples/src/kotlin/org/partiql/examples/util/Main.kt
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ private val examples = mapOf(
EvaluationWithLazyBindings::class.java.simpleName to EvaluationWithLazyBindings(System.out),
ParserErrorExample::class.java.simpleName to ParserErrorExample(System.out),
ParserExample::class.java.simpleName to ParserExample(System.out),
PartialEvaluationRewriterExample::class.java.simpleName to PartialEvaluationRewriterExample(System.out),
PartialEvaluationVisitorTransformExample::class.java.simpleName to PartialEvaluationVisitorTransformExample(System.out),
PreventJoinVisitorExample::class.java.simpleName to PreventJoinVisitorExample(System.out),
SimpleExpressionEvaluation::class.java.simpleName to SimpleExpressionEvaluation(System.out)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ package org.partiql.examples
import org.partiql.examples.util.Example
import java.io.PrintStream

class PartialEvaluationRewriterExampleTest : BaseExampleTest() {
override fun example(out: PrintStream): Example = PartialEvaluationRewriterExample(out)
class PartialEvaluationVisitorTransformExampleTest : BaseExampleTest() {
override fun example(out: PrintStream): Example = PartialEvaluationVisitorTransformExample(out)

override val expected = """
|Original AST:
| (query (plus (lit 1) (lit 1)))
|Rewritten AST:
|Transformed AST:
| (query (lit 2))
|
""".trimMargin()
Expand Down
3 changes: 2 additions & 1 deletion lang/src/org/partiql/lang/ast/AggregateCallSiteListMeta.kt
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,12 @@
package org.partiql.lang.ast

import com.amazon.ion.*
import org.partiql.lang.domains.PartiqlAst

/**
* Contains references to each of the aggregate call-sites in a given [Select].
*/
data class AggregateCallSiteListMeta(val aggregateCallSites: List<CallAgg>): Meta {
data class AggregateCallSiteListMeta(val aggregateCallSites: List<PartiqlAst.Expr.CallAgg>): Meta {
override val tag = TAG

override fun serialize(writer: IonWriter) {
Expand Down
Loading