Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Segment Replication] Add PIT/Scroll compatibility with Segment Replication #7010

Merged

Conversation

dreamer-89
Copy link
Member

Backports #6765 to 2.x

…cation opensearch-project#6644 (opensearch-project#6765)

* Segment Replication - PIT/Scroll compatibility.

This change makes updates to make PIT/Scroll queries compatibile with Segment Replication.
It does this by refcounting files when a new reader is created, and discarding those files after
a reader is closed.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix broken test.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Fix test bug with PIT where snapshotted segments are queried instead of current store state.

Signed-off-by: Marc Handalian <handalm@amazon.com>

* Address review comments and prevent temp file deletion during reader close

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix precommit failure

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Use last committed segment infos reference from replication engine

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Clean up and prevent incref on segment info file copied from primary

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Fix failing test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

---------

Signed-off-by: Marc Handalian <handalm@amazon.com>
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Co-authored-by: Marc Handalian <handalm@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

@dreamer-89
Copy link
Member Author

Unrelated failure

FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':test:fixtures:azure-fixture:composeBuild'.
> Exit-code 1 when calling /usr/bin/docker-compose, stdout: Step 1/5 : FROM ubuntu:18.04

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':test:fixtures:gcs-fixture:composeBuild'.
> Exit-code 1 when calling /usr/bin/docker-compose, stdout: Step 1/11 : FROM ubuntu:18.04
  18.04: Pulling from library/ubuntu

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':test:fixtures:krb5kdc-fixture:composeBuild'.
> Exit-code 1 when calling /usr/bin/docker-compose, stdout: Step 1/9 : FROM ubuntu:14.04
  14.04: Pulling from library/ubuntu

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Suraj Singh <surajrider@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Apr 5, 2023

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testStrictWeightedRoutingWithCustomString
      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all}

@codecov-commenter
Copy link

Codecov Report

Merging #7010 (973f04c) into 2.x (c656949) will decrease coverage by 0.10%.
The diff coverage is 68.19%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #7010      +/-   ##
============================================
- Coverage     70.43%   70.33%   -0.10%     
+ Complexity    59497    59483      -14     
============================================
  Files          4811     4822      +11     
  Lines        285578   285852     +274     
  Branches      41517    41548      +31     
============================================
- Hits         201149   201066      -83     
- Misses        67641    67954     +313     
- Partials      16788    16832      +44     
Impacted Files Coverage Δ
...ch/index/codec/customcodecs/CustomCodecPlugin.java 0.00% <0.00%> (ø)
...h/index/codec/customcodecs/CustomCodecService.java 0.00% <0.00%> (ø)
.../codec/customcodecs/CustomCodecServiceFactory.java 0.00% <0.00%> (ø)
...ustomcodecs/PerFieldMappingPostingFormatCodec.java 0.00% <0.00%> (ø)
.../allocation/allocator/BalancedShardsAllocator.java 91.41% <ø> (-1.02%) ⬇️
...rc/main/java/org/opensearch/index/IndexModule.java 81.92% <ø> (ø)
...c/main/java/org/opensearch/index/IndexService.java 74.49% <0.00%> (+0.39%) ⬆️
...in/java/org/opensearch/indices/IndicesService.java 63.53% <ø> (+2.10%) ⬆️
.../java/org/opensearch/plugins/IndexStorePlugin.java 100.00% <ø> (ø)
...customcodecs/Lucene95CustomStoredFieldsFormat.java 25.00% <25.00%> (ø)
... and 12 more

... and 464 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@dreamer-89 dreamer-89 merged commit 5f19441 into opensearch-project:2.x Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants