Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more index options for streaming job #76

Merged

Conversation

dai-chen
Copy link
Collaborator

@dai-chen dai-chen commented Oct 18, 2023

Description

  1. Add watermark_delay option for MV
  2. Add generic extra_options for other source and sink options
  3. Refactor MV UT with Spark comparePlans() util methods

Documentation

Please see doc here for more details: https://github.com/dai-chen/opensearch-spark/blob/add-more-index-options-for-streaming/docs/index.md#create-index-options

Example

CREATE MATERIALIZED VIEW http_logs_metrics_all
AS
SELECT
  window.start AS startTime,
  COUNT(*) AS count
FROM ds_tables.http_logs
GROUP BY TUMBLE(`@timestamp`, '1 Hour')
WITH (
  auto_refresh = true,
  checkpoint_location = "s3://test/checkpoints/",
  watermark_delay = '1 Minute',
  extra_options = '{"myglue.ds_tables.http_logs": {"maxFilesPerTrigger": "1"}}'
);

Issues Resolved

#25

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@dai-chen dai-chen added the enhancement New feature or request label Oct 18, 2023
@dai-chen dai-chen self-assigned this Oct 18, 2023
@dai-chen dai-chen force-pushed the add-more-index-options-for-streaming branch from a9b894e to 5cd981a Compare October 18, 2023 23:37
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
Signed-off-by: Chen Dai <daichen@amazon.com>
@dai-chen dai-chen force-pushed the add-more-index-options-for-streaming branch from 5cd981a to 101ef1b Compare October 19, 2023 21:39
Signed-off-by: Chen Dai <daichen@amazon.com>
@dai-chen dai-chen marked this pull request as ready for review October 19, 2023 22:06
@dai-chen dai-chen merged commit af210ca into opensearch-project:main Oct 21, 2023
4 checks passed
@dai-chen dai-chen deleted the add-more-index-options-for-streaming branch October 21, 2023 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants