Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable hive support in spark application #1845

Merged
merged 20 commits into from
Jul 19, 2023
Merged

Enable hive support in spark application #1845

merged 20 commits into from
Jul 19, 2023

Conversation

rupal-bq
Copy link
Contributor

@rupal-bq rupal-bq commented Jul 11, 2023 •
edited
Loading

Description

  • Added enableHiveSupport in SparkSession
  • Set flint config in spark session
  • Fix parsing error

Issues Resolved

#1721
#1722

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

rupal-bq added 15 commits June 8, 2023 06:11
@rupal-bq
Initial spark application draft
9c988b0 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Remove temp table
e234a51 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Add license header
25160fd 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Add scalastyle-config and update readme
6f95cf0 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Fix datatype for result and schema
7e02160 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Add test
b72c5a3 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Simplify code using toJSON.collect.toList
1883b25 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Add example in readme
3261028 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Fix triple quotes issue
f815709 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Update method name and description
ec08d0b 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
65e0b1c 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Add applicationId
959d5b2 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
82c70f2 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
f9970b2 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Enable Hive Support
df9c4c0 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
6a97c19 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
penghuo
penghuo previously approved these changes Jul 11, 2023
View reviewed changes
@penghuo
Copy link
Collaborator

penghuo commented Jul 11, 2023

ignore all the IT. not related to this PR.

@Yury-Fridlyand
Copy link
Collaborator

You can rebase your branch to make CI pass

rupal-bq added 2 commits July 12, 2023 11:31
@rupal-bq
Set flint config
35526a8 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
8e787a0 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@codecov
Copy link

codecov bot commented Jul 12, 2023 •
edited
Loading

Codecov Report

Merging #1845 (d2ff0eb) into main (8c8e08c) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #1845   +/-   ##
=========================================
  Coverage     97.39%   97.39%           
  Complexity     4608     4608           
=========================================
  Files           401      401           
  Lines         11408    11408           
  Branches        843      843           
=========================================
  Hits          11111    11111           
  Misses          290      290           
  Partials          7        7
Flag Coverage Δ
sql-engine 97.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

rupal-bq added 2 commits July 14, 2023 12:28
@rupal-bq
Fix json parsing error
191964b 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Merge branch 'main' into spark-application
d2ff0eb 
Signed-off-by: Rupal Mahajan <maharup@amazon.com>
@rupal-bq
Copy link
Contributor Author

bwc test failure is not related to this PR

dai-chen
dai-chen approved these changes Jul 19, 2023
View reviewed changes
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@rupal-bq rupal-bq merged commit a34ac9b into opensearch-project:main Jul 19, 2023
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 20, 2023
@rupal-bq @MitchellGale
Enable hive support in spark application (opensearch-project#1845)
632feed 
* Initial spark application draft

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Remove temp table

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add license header

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add scalastyle-config and update readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix datatype for result and schema

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add test

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Simplify code using toJSON.collect.toList

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add example in readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix triple quotes issue

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update method name and description

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add applicationId

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Enable Hive Support

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Set flint config

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix json parsing error

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

---------

Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 20, 2023
@rupal-bq @MitchellGale
Enable hive support in spark application (opensearch-project#1845)
9e87205 
* Initial spark application draft

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Remove temp table

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add license header

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add scalastyle-config and update readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix datatype for result and schema

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add test

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Simplify code using toJSON.collect.toList

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add example in readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix triple quotes issue

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update method name and description

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add applicationId

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Enable Hive Support

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Set flint config

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix json parsing error

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

---------

Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
matthewryanwells pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 20, 2023
@rupal-bq @matthewryanwells
Enable hive support in spark application (opensearch-project#1845)
99e6f0a 
* Initial spark application draft

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Remove temp table

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add license header

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add scalastyle-config and update readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix datatype for result and schema

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add test

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Simplify code using toJSON.collect.toList

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add example in readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix triple quotes issue

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update method name and description

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add applicationId

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Enable Hive Support

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Set flint config

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix json parsing error

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

---------

Signed-off-by: Rupal Mahajan <maharup@amazon.com>
MitchellGale pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Jul 28, 2023
@rupal-bq @MitchellGale
Enable hive support in spark application (opensearch-project#1845)
661cec6 
* Initial spark application draft

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Remove temp table

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add license header

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add scalastyle-config and update readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix datatype for result and schema

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add test

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Simplify code using toJSON.collect.toList

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add example in readme

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix triple quotes issue

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Update method name and description

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Add applicationId

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Enable Hive Support

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Set flint config

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

* Fix json parsing error

Signed-off-by: Rupal Mahajan <maharup@amazon.com>

---------

Signed-off-by: Rupal Mahajan <maharup@amazon.com>
Signed-off-by: Mitchell Gale <Mitchell.Gale@improving.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dai-chen dai-chen dai-chen approved these changes

@anirudha anirudha anirudha approved these changes

@penghuo penghuo penghuo left review comments

@pjfitzgibbons pjfitzgibbons Awaiting requested review from pjfitzgibbons

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vmmusings vmmusings Awaiting requested review from vmmusings

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rupal-bq @penghuo @Yury-Fridlyand @anirudha @dai-chen