Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE]Setup Spark Connector Module #1721

Open
vmmusings opened this issue Jun 8, 2023 · 0 comments
Open

[FEATURE]Setup Spark Connector Module #1721

vmmusings opened this issue Jun 8, 2023 · 0 comments
Assignees
@vmmusings
Labels
enhancement New feature or request

Comments

@vmmusings
Copy link
Member

vmmusings commented Jun 8, 2023 •
edited by rupal-bq
Loading

Is your feature request related to a problem?
Add spark datasource support

  1. Setup Basic Spark Connector Module.
  2. Support EMR connector
  3. Support Livy connector

Proposed Approches:

  1. Synchronous
    image (2)

  2. Asynchronous
    image (3)

Comparison

  Synchronous Async with Opensearch index
Long Execution/Waiting Time Yes (min time for query ~30sec) No
Query limitations Queries which need timeout more than 5 mins won't work NA
Index management required? Yes Yes
@vmmusings vmmusings added the enhancement New feature or request label Jun 8, 2023
@vmmusings vmmusings self-assigned this Jun 8, 2023
@vmmusings vmmusings linked a pull request Jun 8, 2023 that will close this issue
Create Spark Connector #1720
Closed
6 tasks
@vmmusings vmmusings mentioned this issue Jun 8, 2023
Closed
6 tasks
This was referenced Jun 26, 2023
Merged
Merged
@rupal-bq rupal-bq mentioned this issue Jul 14, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vmmusings vmmusings

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Create Spark Connector vmmusings/sql
2 participants
@mengweieric @vmmusings