-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Flint Spark index management API #1636
Add Flint Spark index management API #1636
Conversation
Signed-off-by: Chen Dai <daichen@amazon.com>
Codecov Report
@@ Coverage Diff @@
## feature/flint opensearch-project/sql#1636 +/- ##
================================================
Coverage 97.19% 97.19%
Complexity 4107 4107
================================================
Files 371 371
Lines 10464 10464
Branches 706 706
================================================
Hits 10170 10170
Misses 287 287
Partials 7 7
Flags with carried forward coverage won't be shown. Click here to find out more. |
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
flint/flint-spark-integration/src/main/scala/org/opensearch/flint/spark/FlintSpark.scala
Show resolved
Hide resolved
/** | ||
* Flint configurations in Spark. TODO: shared with Flint data source config? | ||
*/ | ||
val FLINT_INDEX_STORE_LOCATION = "spark.flint.indexstore.location" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
QQ, should we put all the flint spark related configuration in here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I think we should put all together here or a new FlintSparkConf
class later.
flint/flint-spark-integration/src/main/scala/org/opensearch/flint/spark/FlintSpark.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Chen Dai <daichen@amazon.com>
Description
FlintSpark
andFlintSparkSkippingIndex
PartitionSkippingStrategy
andFlintClient.deleteIndex()
for integration testTODO
Implement Partition index building once Flint batch writer is ready
API Example
Please find more details in Flint doc, IT and detailed design in opensearch-project/opensearch-spark#2.
Issues Resolved
opensearch-project/opensearch-spark#2
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.