Integrating Parser with Planner #49

gaurav274 · 2020-01-31T02:40:32Z

Catalog merged with the planner.
Logical operators added and supported for select query
Logical Plan tree created
Tuple Value Expression changed and the corresponding test cases fixed.

ToDo:
Still need to add robust test cases for 1,2,3

- Moved classes DataFrameMetadata and DataFrameColumn to database models - Added a database configuration file and a Base Model class - Added methods for retrieving bindings and metadata object

…ee built using operators. 3. Util functionality for binding tables, columns with catalog.

…that

jarulraj · 2020-01-31T04:39:13Z

src/catalog/catalog_manager.py

-        rows = [row_1]
-
-        append_rows(dataset_catalog_entry, rows)
+        init_db()


Is the output_url not being used?

jarulraj · 2020-01-31T04:39:29Z

src/catalog/catalog_manager.py

-
-        append_rows(dataset_catalog_entry, rows)
+        init_db()
+        # # Construct output location


Can we remove the commented out code?

jarulraj · 2020-01-31T04:39:59Z

src/catalog/catalog_manager.py

+                DataFrameSchema(metadata.get_name(), df_columns))
+        return metadata
+
+    # def create_dataset(self, dataset_name: str):


Can this function be removed?

jarulraj · 2020-01-31T04:40:35Z

src/catalog/database.py

+    """
+    Add and try to flush.
+    """
+


jarulraj · 2020-01-31T04:41:20Z

src/catalog/database.py

+from sqlalchemy.exc import DatabaseError
+from sqlalchemy_utils import database_exists, create_database, drop_database
+
+engine = create_engine(SQLALCHEMY_DATABASE_URI)


Should these variables be global?

Can we move the database uri to configuration.yml?

jarulraj · 2020-01-31T04:41:48Z

src/catalog/database.py

+            db_session.rollback()
+
+
+BaseModel = declarative_base(cls=CustomBase, constructor=None)


Global variable?

jarulraj · 2020-01-31T04:42:21Z

src/catalog/df_column.py

+from src.utils.logging_manager import LoggingManager
+
+
+class DataframeColumnType(Enum):


Why not reuse the ColumnType itself?

jarulraj · 2020-01-31T04:43:22Z

src/catalog/models/df_column.py

+        return self._is_nullable
+
+    def get_array_dimensions(self):
+        return json.loads(self._array_dimensions)


Do we need json to do this?

jarulraj · 2020-01-31T04:44:10Z

src/catalog/rough.ipynb

@@ -0,0 +1,181 @@
+{
+ "cells": [


Should we remove this notebook? Should we add an ignore line in .gitignore for notebooks?

jarulraj · 2020-01-31T04:44:53Z

test/catalog/test_catalog_manager.py

-        x.create_dataset("foo")
-        x.create_dataset("bar")
-        x.create_dataset("baz")
+        # x.create_dataset("foo")


Can we remove this code? Maybe add some other programmatic tests?

SanmathiK and others added 30 commits December 4, 2019 12:58

Sample test file

e0645b0

Testing visitFullColumnName function

b380f11

Documentation

c1bc54b

Added Unit Test Case for visitLogicalExpression

fde9ca6

Added Unit Test Case for visitTableName

28b3472

Added Tests for visitStringLiteral function

a5a632d

Added unit test for query_parser: visitConstant

1202f2d

Testing visitQuerySpecification

8989039

Fixing pycodestyle errors

a52786d

Fixing pycodestyle

fce764c

Testing SelectStatement

169ddde

Merge branch 'master' of https://github.com/KokareIITP/Eva

73acaa4

Added unit test for class TableRef

151ecc6

Fixing indentation errors

b93773a

Fixed continuation line under-indented for visual indent

1c08814

Cleaning Up

d6806ad

Update README

c7b2d47

Edit ReadMe

7aa0269

temp commit

3fb3f88

planner nodes updated

bdc4164

Plan generator and convertor

7f1ea13

Merged upstream

957166c

File deleted. renaming to operators

f39fa0a

parser to logical operator

f99afdd

Formatting query_planner

e8a9aed

formatting changes

193b474

Formatting changes

5f8b985

Catalog

603429b

- Moved classes DataFrameMetadata and DataFrameColumn to database models - Added a database configuration file and a Base Model class - Added methods for retrieving bindings and metadata object

Fixed test case using mock

15671b9

Added mock to environment.yml

d321493

gaurav274 added 14 commits January 30, 2020 16:46

Added support for catalog bindings in the tuple expression

1b383b6

1. Parsed statement converted to Logical Operator. 2. Logical plan tr…

9807506

…ee built using operators. 3. Util functionality for binding tables, columns with catalog.

Directory renamed to optimizer

902a267

Obsolete Files removed

8602bdc

Removed Obsolete files

0d4f6df

ToDo: Generate Physical plan without any optimizer

2ef03e9

merged PR

d36053e

Checkpoint: minor fixes

c7f26bb

Files renamed

580a45b

Ran Formatter

de0171c

TupleExpression replaced by ConstantExpression

043d044

Bug fixed

7d29016

No more relevant test cases

6f207c8

Skipping cases beacuse SeqScan API changed; Will fix after we update …

5993376

…that

gaurav274 requested review from jarulraj and sanjanag January 31, 2020 02:40

jarulraj reviewed Jan 31, 2020

View reviewed changes

src/catalog/catalog_manager.py

rows = [row_1]

append_rows(dataset_catalog_entry, rows)

init_db()

Copy link

Member

jarulraj Jan 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the output_url not being used?

jarulraj reviewed Jan 31, 2020

View reviewed changes

src/catalog/database.py

"""

Add and try to flush.

"""

Copy link

Member

jarulraj Jan 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Space?

jarulraj reviewed Jan 31, 2020

View reviewed changes

src/catalog/database.py

db_session.rollback()

BaseModel = declarative_base(cls=CustomBase, constructor=None)

Copy link

Member

jarulraj Jan 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Global variable?

jarulraj reviewed Jan 31, 2020

View reviewed changes

jarulraj closed this Jan 31, 2020

jarulraj reopened this Jan 31, 2020

jarulraj merged commit 5993376 into georgia-tech-db:master Feb 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrating Parser with Planner #49

Integrating Parser with Planner #49

gaurav274 commented Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

jarulraj Jan 31, 2020

		db_session.rollback()


		BaseModel = declarative_base(cls=CustomBase, constructor=None)

		from src.utils.logging_manager import LoggingManager


		class DataframeColumnType(Enum):

Integrating Parser with Planner #49

Integrating Parser with Planner #49

Conversation

gaurav274 commented Jan 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment