Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding json_table function #1142

Merged
merged 38 commits into from
Aug 16, 2022
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2ac0b17
starting on jsonpath logic
Jul 29, 2022
07d307b
:(
Jul 29, 2022
0a89181
trying out jsonpath parsing package
Jul 31, 2022
da2958b
[ga-format-pr] Run ./format_repo.sh to fix formatting
jcor11599 Jul 31, 2022
5a3e53e
make new JSONTable node
Aug 1, 2022
db01268
adding partitions, and starting to create in-memory table
Aug 1, 2022
9a687c9
merge
Aug 1, 2022
23aeb31
Merge branch 'main' into james/json_table
Aug 2, 2022
2982d30
working
Aug 2, 2022
19477b5
fixing todo
Aug 2, 2022
f1314c8
Merge branch 'main' into james/json_table
Aug 2, 2022
855b347
Merge branch 'main' into james/json_table
Aug 8, 2022
a51c1db
Merge branch 'main' into james/json_table
Aug 10, 2022
6d93e54
trying stuff
Aug 11, 2022
5390ab1
Merge branch 'main' into james/json_table
Aug 11, 2022
9c2f480
working so far
Aug 11, 2022
2f76cb2
parse expr
Aug 11, 2022
cfb9a6a
bump
Aug 11, 2022
5dfcc74
moving sqlparser code out of plan
Aug 12, 2022
05f39a7
move logic to rowiter
Aug 12, 2022
ab28f59
adding a bad test
Aug 12, 2022
03cedc8
a
Aug 12, 2022
50dedb6
a
Aug 12, 2022
47d4512
Merge branch 'main' into james/json_table
Aug 14, 2022
040422d
i did this yesterday and it didn't work, today is a different day
Aug 14, 2022
5b51930
removing unnecessary setup
Aug 15, 2022
01de00d
bumping
Aug 15, 2022
31d42ea
fix comment
Aug 15, 2022
c28fee8
more tests
Aug 15, 2022
bd44c84
adding column test
Aug 15, 2022
55dfbd3
merge with main
Aug 15, 2022
a9e1cdf
passing in row
Aug 15, 2022
a758d4d
removing comment
Aug 16, 2022
56dce6e
Merge branch 'main' into james/json_table
Aug 16, 2022
f4966c3
moving logic into next
Aug 16, 2022
07b674a
avoid precision problem
Aug 16, 2022
80aca34
adding test for null
Aug 16, 2022
fd72a9e
use single quote instead
Aug 16, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions enginetest/enginetests.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,13 @@ func TestJoinQueries(t *testing.T, harness Harness) {
}
}

func TestJSONTableQueries(t *testing.T, harness Harness) {
harness.Setup(setup.MydbData, setup.Pk_tablesData, setup.Json_table_tablesData)
for _, tt := range queries.JSONTableQueryTests {
TestQuery(t, harness, tt.Query, tt.Expected, tt.ExpectedColumns, nil)
}
}

// TestInfoSchemaPrepared runs tests of the information_schema database
func TestInfoSchemaPrepared(t *testing.T, harness Harness) {
harness.Setup(setup.MydbData, setup.MytableData, setup.Fk_tblData, setup.FooData)
Expand Down
2 changes: 2 additions & 0 deletions enginetest/evaluation.go
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,8 @@ func WidenRow(sch sql.Schema, row sql.Row) sql.Row {
vw = uint64(x)
case float32:
vw = float64(x)
case float64:
vw = float64(float32(x))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are rounding errors when using float64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There really shouldn't be, this is a sign of a bug

Floats should round-trip with no precision loss

Does it come from parsing JSON?

default:
vw = v
}
Expand Down
5 changes: 5 additions & 0 deletions enginetest/memory_engine_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,11 @@ func TestJoinQueries(t *testing.T) {
enginetest.TestJoinQueries(t, enginetest.NewMemoryHarness("simple", 1, testNumPartitions, true, nil))
}

// TestJSONQueries runs the canonical test queries against a single threaded index enabled harness.
func TestJSONTableQueries(t *testing.T) {
enginetest.TestJSONTableQueries(t, enginetest.NewMemoryHarness("simple", 1, testNumPartitions, true, nil))
}

// Convenience test for debugging a single query. Unskip and set to the desired query.
func TestSingleQuery(t *testing.T) {
t.Skip()
Expand Down
116 changes: 116 additions & 0 deletions enginetest/queries/json_table_queries.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
// Copyright 2022 Dolthub, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package queries

import (
"github.com/dolthub/go-mysql-server/sql"
)

var JSONTableQueryTests = []QueryTest{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need some tests joining json tables to physical tables

If they don't work, add skipped tests

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tests where the expression for json data comes from a column, not a string literal

{
Query: "SELECT * FROM JSON_TABLE('[{\"a\":1},{\"a\":2}]',\"$[*]\" COLUMNS(x varchar(100) path \"$.a\")) as tt;",
Expected: []sql.Row{
{"1"},
{"2"},
},
},
{
Query: "SELECT * FROM JSON_TABLE('[{\"a\":1, \"b\":2},{\"a\":3, \"b\":4}]',\"$[*]\" COLUMNS(x int path \"$.a\", y int path \"$.b\")) as tt;",
Expected: []sql.Row{
{1, 2},
{3, 4},
},
},
{
Query: "SELECT * FROM JSON_TABLE('[{\"a\":1.123, \"b\":2.234},{\"a\":3.345, \"b\":4.456}]',\"$[*]\" COLUMNS(x float path \"$.a\", y float path \"$.b\")) as tt;",
Expected: []sql.Row{
{1.123, 2.234},
{3.345, 4.456},
},
},
{
Query: "SELECT * FROM JSON_TABLE(concat('[{},','{}]'),\"$[*]\" COLUMNS(x varchar(100) path \"$.a\",y varchar(100) path \"$.b\")) as t;",
Expected: []sql.Row{
{nil, nil},
{nil, nil},
},
},
{
Query: "select * from JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t1 join JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t2;",
Expected: []sql.Row{
{1, 1},
{1, 2},
{2, 1},
{2, 2},
},
},
{
Query: "select * from JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t1 join one_pk order by x;",
Expected: []sql.Row{
{1, 0, 0, 1, 2, 3, 4},
{1, 1, 10, 11, 12, 13, 14},
{1, 2, 20, 21, 22, 23, 24},
{1, 3, 30, 31, 32, 33, 34},
{2, 0, 0, 1, 2, 3, 4},
{2, 1, 10, 11, 12, 13, 14},
{2, 2, 20, 21, 22, 23, 24},
{2, 3, 30, 31, 32, 33, 34},
},
},
{
Query: "select * from one_pk join JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t1 order by x;",
Expected: []sql.Row{
{0, 0, 1, 2, 3, 4, 1},
{1, 10, 11, 12, 13, 14, 1},
{2, 20, 21, 22, 23, 24, 1},
{3, 30, 31, 32, 33, 34, 1},
{0, 0, 1, 2, 3, 4, 2},
{1, 10, 11, 12, 13, 14, 2},
{2, 20, 21, 22, 23, 24, 2},
{3, 30, 31, 32, 33, 34, 2},
},
},
{
Query: "select * from JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t1 union select * from JSON_TABLE('[{\"b\":3},{\"b\":4}]', \"$[*]\" COLUMNS(y int path \"$.b\")) as t2",
Expected: []sql.Row{
{1},
{2},
{3},
{4},
},
},
{
Query: "select * from one_pk where pk in (select x from JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) as t)",
Expected: []sql.Row{
{1, 10, 11, 12, 13, 14},
{2, 20, 21, 22, 23, 24},
},
},
{
Query: "select * from JSON_TABLE('[{\"a\":1},{\"a\":2}]', \"$[*]\" COLUMNS(x int path \"$.a\")) t1 where x in (select y from JSON_TABLE('[{\"b\":1},{\"b\":100}]', \"$[*]\" COLUMNS(y int path \"$.b\")) as t2)",
Expected: []sql.Row{
{1},
},
},
{
Query: "SELECT * FROM JSON_TABLE((select t from json_table_tables),\"$[*]\" COLUMNS(i int path \"$.a\", j int path \"$.b\", k int path \"$.c\", l int path \"$.d\")) as tt;",
Expected: []sql.Row{
{1, nil, nil, nil},
{nil, 2, nil, nil},
{nil, nil, 3, nil},
{nil, nil, nil, 4},
},
},
}
11 changes: 11 additions & 0 deletions enginetest/scriptgen/setup/scripts/json_table_tables
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

exec
CREATE TABLE `json_table_tables` (
`t` varchar(100)
)
----

exec
insert into json_table_tables values
('[{"a": 1},{"b": 2},{"c": 3},{"d": 4}]');
----
5 changes: 5 additions & 0 deletions enginetest/scriptgen/setup/setup_data.sg.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/dolthub/go-mysql-server
require (
github.com/cespare/xxhash v1.1.0
github.com/dolthub/sqllogictest/go v0.0.0-20201107003712-816f3ae12d81
github.com/dolthub/vitess v0.0.0-20220815191244-a43b8493fb04
github.com/dolthub/vitess v0.0.0-20220815214835-68adfbec4880
github.com/go-kit/kit v0.10.0
github.com/go-sql-driver/mysql v1.6.0
github.com/gocraft/dbr/v2 v2.7.2
Expand Down
10 changes: 2 additions & 8 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -55,14 +55,8 @@ github.com/dolthub/jsonpath v0.0.0-20210609232853-d49537a30474 h1:xTrR+l5l+1Lfq0
github.com/dolthub/jsonpath v0.0.0-20210609232853-d49537a30474/go.mod h1:kMz7uXOXq4qRriCEyZ/LUeTqraLJCjf0WVZcUi6TxUY=
github.com/dolthub/sqllogictest/go v0.0.0-20201107003712-816f3ae12d81 h1:7/v8q9XGFa6q5Ap4Z/OhNkAMBaK5YeuEzwJt+NZdhiE=
github.com/dolthub/sqllogictest/go v0.0.0-20201107003712-816f3ae12d81/go.mod h1:siLfyv2c92W1eN/R4QqG/+RjjX5W2+gCTRjZxBjI3TY=
github.com/dolthub/vitess v0.0.0-20220809164912-4bb30c793f1b h1:KDimfMuvlB0be3XhBBBF5QWIJGzazleccRlHyf/dXX8=
github.com/dolthub/vitess v0.0.0-20220809164912-4bb30c793f1b/go.mod h1:oVFIBdqMFEkt4Xz2fzFJBNtzKhDEjwdCF0dzde39iKs=
github.com/dolthub/vitess v0.0.0-20220810172629-f4bdb4a721c0 h1:u8yIimnhg5CHQZlBoJxfV+yezvNEmtgXUsmZvvlJ6uo=
github.com/dolthub/vitess v0.0.0-20220810172629-f4bdb4a721c0/go.mod h1:oVFIBdqMFEkt4Xz2fzFJBNtzKhDEjwdCF0dzde39iKs=
github.com/dolthub/vitess v0.0.0-20220811180048-dec9dd483049 h1:rw68oRFvBt9QdaGrNCEr2TE5NCYax7JfytLQlGsvTg8=
github.com/dolthub/vitess v0.0.0-20220811180048-dec9dd483049/go.mod h1:oVFIBdqMFEkt4Xz2fzFJBNtzKhDEjwdCF0dzde39iKs=
github.com/dolthub/vitess v0.0.0-20220815191244-a43b8493fb04 h1:apUYKsKc4DgjYkxV+ZfE/B4MdRXbiiwJh7hQ18FeAto=
github.com/dolthub/vitess v0.0.0-20220815191244-a43b8493fb04/go.mod h1:oVFIBdqMFEkt4Xz2fzFJBNtzKhDEjwdCF0dzde39iKs=
github.com/dolthub/vitess v0.0.0-20220815214835-68adfbec4880 h1:f/t/cgRA43r71qqbQUpcd772hKRuK/m1Nz2fo7f/C7o=
github.com/dolthub/vitess v0.0.0-20220815214835-68adfbec4880/go.mod h1:oVFIBdqMFEkt4Xz2fzFJBNtzKhDEjwdCF0dzde39iKs=
github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
github.com/eapache/go-resiliency v1.1.0/go.mod h1:kFI+JgMyC7bLPUVY133qvEBtVayf5mFgVsvEsIPBvNs=
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21/go.mod h1:+020luEh2TKB4/GOp8oxxtq0Daoen/Cii55CzbTV6DU=
Expand Down
22 changes: 22 additions & 0 deletions sql/parse/parse.go
Original file line number Diff line number Diff line change
Expand Up @@ -2605,6 +2605,9 @@ func tableExprToTable(
case *sqlparser.JoinTableExpr:
return joinTableExpr(ctx, t)

case *sqlparser.JSONTableExpr:
return jsonTableExpr(ctx, t)

case *sqlparser.ParenTableExpr:
if len(t.Exprs) == 1 {
switch j := t.Exprs[0].(type) {
Expand Down Expand Up @@ -2661,6 +2664,25 @@ func joinTableExpr(ctx *sql.Context, t *sqlparser.JoinTableExpr) (sql.Node, erro
}
}

func jsonTableExpr(ctx *sql.Context, t *sqlparser.JSONTableExpr) (sql.Node, error) {
data, err := ExprToExpression(ctx, t.Data)
if err != nil {
return nil, err
}

paths := make([]string, len(t.Spec.Columns))
for i, col := range t.Spec.Columns {
paths[i] = col.Type.Path
}

sch, err := TableSpecToSchema(ctx, t.Spec)
if err != nil {
return nil, err
}

return plan.NewJSONTable(ctx, data, t.Path, paths, t.Alias.String(), sch)
}

func whereToFilter(ctx *sql.Context, w *sqlparser.Where, child sql.Node) (*plan.Filter, error) {
c, err := ExprToExpression(ctx, w.Expr)
if err != nil {
Expand Down
Loading