Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] PartiallyCachedShardAllocationIntegTests testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown failing #91800

Closed
dimitris-athanasiou opened this issue Nov 22, 2022 · 9 comments
Closed

[CI] PartiallyCachedShardAllocationIntegTests testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown failing #91800

dimitris-athanasiou opened this issue Nov 22, 2022 · 9 comments
Assignees
@idegtiarenko @DaveCTurner
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@dimitris-athanasiou
Copy link
Contributor

Build scan:
https://gradle-enterprise.elastic.co/s/yxdhlgzs42t74/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests/testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown

Reproduction line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown" -Dtests.seed=B26EDDAAD545FA23 -Dtests.locale=fr-LU -Dtests.timezone=Australia/Eucla -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests&tests.test=testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown

Failure excerpt:

java.lang.AssertionError: balanced across [node_s3, node_s4] in cluster uuid: g6F-mWO8TtCnzyD046ozoQ [committed: false]
version: 150
state uuid: f1ulFeM-SWubxzBqr_x7Gw
from_diff: false
meta data version: 0
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{BauHnEvMQ0y2dpsxcgtXhQ,FNxIhvGuS0GZX8I3NCpYEg,p1h7-2W-RTifRQPueIzsVQ}
      last_accepted_config: VotingConfiguration{BauHnEvMQ0y2dpsxcgtXhQ,FNxIhvGuS0GZX8I3NCpYEg,p1h7-2W-RTifRQPueIzsVQ}
      voting tombstones: []
metadata customs:
   index-graveyard: IndexGraveyard[[]]
nodes: 
   {node_s1}{FNxIhvGuS0GZX8I3NCpYEg}{57478pZYTTW3jSUTHwlsvg}{node_s1}{127.0.0.1}{127.0.0.1:26593}{cdfhilmrstw}{xpack.installed=true}
   {node_s4}{uBXJWw2QR-KCV5TVKuiAgg}{R9yDbEOuSHCZoUTWunn4NQ}{node_s4}{127.0.0.1}{127.0.0.1:26594}{d}{xpack.installed=true}
   {node_s2}{p1h7-2W-RTifRQPueIzsVQ}{A-UGdHN2STSq0oQLM4Jxpg}{node_s2}{127.0.0.1}{127.0.0.1:26592}{cdfhilmrstw}{xpack.installed=true}
   {node_s0}{BauHnEvMQ0y2dpsxcgtXhQ}{lcivK9g1SkaFuO7NdSyH0w}{node_s0}{127.0.0.1}{127.0.0.1:26591}{cdfhilmrstw}{xpack.installed=true}, master
   {node_s3}{9JoDQNZtQvS7Yl1BqcLQfQ}{4Hkmjh2-SqO7q212LV_0zA}{node_s3}{127.0.0.1}{127.0.0.1:26595}{d}{xpack.installed=true}
routing_table (version 73):
-- index [[.snapshot-blob-cache/oSN0ikv6QmyR9kstVX1OdA]]
----shard_id [.snapshot-blob-cache][0]
--------[.snapshot-blob-cache][0], node[BauHnEvMQ0y2dpsxcgtXhQ], [P], s[STARTED], a[id=rfHV99GWTj694MLLvjn74A], failed_attempts[0]
--------[.snapshot-blob-cache][0], node[FNxIhvGuS0GZX8I3NCpYEg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=mFrs3JFgR4eHMV3CXBMN9A], unassigned_info[[reason=INDEX_CREATED], at[2022-11-22T11:45:58.321Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]

-- index [[mmqwiqmzwb/dFYFWDNrQYqDXRiqyaLxqw]]
----shard_id [mmqwiqmzwb][0]
--------[mmqwiqmzwb][0], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=ViZVePKVTzGZofmvgUjaVw], failed_attempts[0], expected_shard_size[7383]
----shard_id [mmqwiqmzwb][1]
--------[mmqwiqmzwb][1], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=iKdA5z4CT7qT-K38NVs9tg], failed_attempts[0], expected_shard_size[7407]
----shard_id [mmqwiqmzwb][2]
--------[mmqwiqmzwb][2], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=f0vRmhnxQeOKMJ8H0iIa4w], failed_attempts[0], expected_shard_size[5110]
----shard_id [mmqwiqmzwb][3]
--------[mmqwiqmzwb][3], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=7zvD2FxOSVSh_aC0DJFL1Q], failed_attempts[0], expected_shard_size[5182]
----shard_id [mmqwiqmzwb][4]
--------[mmqwiqmzwb][4], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=a57riAlZR42Wb6uNXel2tQ], failed_attempts[0], expected_shard_size[15805]
----shard_id [mmqwiqmzwb][5]
--------[mmqwiqmzwb][5], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=oJh9FdrTSsy_lnQgMIegnw], failed_attempts[0], expected_shard_size[225]
----shard_id [mmqwiqmzwb][6]
--------[mmqwiqmzwb][6], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=37w3sOE7Tay0BS5Q9KQWNQ], failed_attempts[0], expected_shard_size[5110]
----shard_id [mmqwiqmzwb][7]
--------[mmqwiqmzwb][7], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=VkymzL97SqiixdInBAYyWQ], failed_attempts[0], expected_shard_size[12338]
----shard_id [mmqwiqmzwb][8]
--------[mmqwiqmzwb][8], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=rxaohcUiTnm4ZcB7BclhEw], failed_attempts[0], expected_shard_size[5813]
----shard_id [mmqwiqmzwb][9]
--------[mmqwiqmzwb][9], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=VpAmimXERsaE-gjHp4XNbA], failed_attempts[0], expected_shard_size[5110]

routing_nodes:
-----node_id[9JoDQNZtQvS7Yl1BqcLQfQ][V]
--------[mmqwiqmzwb][0], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=ViZVePKVTzGZofmvgUjaVw], failed_attempts[0], expected_shard_size[7383]
--------[mmqwiqmzwb][2], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=f0vRmhnxQeOKMJ8H0iIa4w], failed_attempts[0], expected_shard_size[5110]
--------[mmqwiqmzwb][4], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=a57riAlZR42Wb6uNXel2tQ], failed_attempts[0], expected_shard_size[15805]
--------[mmqwiqmzwb][6], node[9JoDQNZtQvS7Yl1BqcLQfQ], [P], s[STARTED], a[id=37w3sOE7Tay0BS5Q9KQWNQ], failed_attempts[0], expected_shard_size[5110]
-----node_id[BauHnEvMQ0y2dpsxcgtXhQ][V]
--------[.snapshot-blob-cache][0], node[BauHnEvMQ0y2dpsxcgtXhQ], [P], s[STARTED], a[id=rfHV99GWTj694MLLvjn74A], failed_attempts[0]
-----node_id[FNxIhvGuS0GZX8I3NCpYEg][V]
--------[.snapshot-blob-cache][0], node[FNxIhvGuS0GZX8I3NCpYEg], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=mFrs3JFgR4eHMV3CXBMN9A], unassigned_info[[reason=INDEX_CREATED], at[2022-11-22T11:45:58.321Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]
-----node_id[uBXJWw2QR-KCV5TVKuiAgg][V]
--------[mmqwiqmzwb][1], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=iKdA5z4CT7qT-K38NVs9tg], failed_attempts[0], expected_shard_size[7407]
--------[mmqwiqmzwb][3], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=7zvD2FxOSVSh_aC0DJFL1Q], failed_attempts[0], expected_shard_size[5182]
--------[mmqwiqmzwb][5], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=oJh9FdrTSsy_lnQgMIegnw], failed_attempts[0], expected_shard_size[225]
--------[mmqwiqmzwb][7], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=VkymzL97SqiixdInBAYyWQ], failed_attempts[0], expected_shard_size[12338]
--------[mmqwiqmzwb][8], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=rxaohcUiTnm4ZcB7BclhEw], failed_attempts[0], expected_shard_size[5813]
--------[mmqwiqmzwb][9], node[uBXJWw2QR-KCV5TVKuiAgg], [P], s[STARTED], a[id=VpAmimXERsaE-gjHp4XNbA], failed_attempts[0], expected_shard_size[5110]
-----node_id[p1h7-2W-RTifRQPueIzsVQ][V]
---- unassigned

  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown(PartiallyCachedShardAllocationIntegTests.java:325)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@dimitris-athanasiou dimitris-athanasiou added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Nov 22, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Nov 22, 2022
@original-brownbear
Copy link
Member

@DaveCTurner this looks like it might be an issue with the new allocation decider? It's kind of strange how we get a lopsided allocation here for the new mount across two empty nodes here now, maybe you have an idea?

@original-brownbear original-brownbear removed their assignment Nov 22, 2022
@DaveCTurner DaveCTurner self-assigned this Nov 22, 2022
@pxsalehi
Copy link
Member

pxsalehi commented Dec 6, 2022

Failed on 8.6: https://gradle-enterprise.elastic.co/s/n2fya3bzklnog/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests/testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown?top-execution=1

@pxsalehi pxsalehi mentioned this issue Dec 7, 2022
Closed
@ywangd ywangd mentioned this issue Dec 11, 2022
Merged
@astefan
Copy link
Contributor

astefan commented Dec 23, 2022

Another failure here and it doesn't reproduce:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown" -Dtests.seed=3CB0B2A7DEE4BD03 -Dtests.locale=sr-RS -Dtests.timezone=US/Arizona -Druntime.java=17

org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests > testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown FAILED
    java.lang.AssertionError: balanced across [node_s2, node_s3] in cluster uuid: CxH-NNQVTHubDdrHxz2oGg [committed: false]
    version: 170
    state uuid: 2uvN4TU2T5GUq65G4pqQTw
    from_diff: false
    meta data version: 0
       coordination_metadata:
          term: 1
          last_committed_config: VotingConfiguration{eBGx5PwCQ42xJq8vqRKqyw}
          last_accepted_config: VotingConfiguration{eBGx5PwCQ42xJq8vqRKqyw}
          voting tombstones: []
    metadata customs:
       index-graveyard: IndexGraveyard[[]]
    nodes: 
       {node_s0}{tpz5rx_gRRa3OjkJI6mloA}{qT7Mc2ZXQluuBDHZW2S6Vg}{node_s0}{127.0.0.1}{127.0.0.1:17141}{cdfhilmrstw}{xpack.installed=true}
       {node_s1}{eBGx5PwCQ42xJq8vqRKqyw}{cVNMr6SlRqeswDpShVX_Ug}{node_s1}{127.0.0.1}{127.0.0.1:17142}{cdfhilmrstw}{xpack.installed=true}, master
       {node_s2}{RyaaHVHRQne1t5gJMiOrxQ}{TWCCvpvyQM-6EIdDpCtl9A}{node_s2}{127.0.0.1}{127.0.0.1:17144}{d}{xpack.installed=true}
       {node_s3}{BQxMyRTARK--skKRDnj97A}{qnkuL_CrQDatm8IHAC_i6w}{node_s3}{127.0.0.1}{127.0.0.1:17143}{d}{xpack.installed=true}
    routing_table (version 92):
    -- index [[.snapshot-blob-cache/KshEuuC8THelJwR2-tU8XQ]]
    ----shard_id [.snapshot-blob-cache][0]
    --------[.snapshot-blob-cache][0], node[eBGx5PwCQ42xJq8vqRKqyw], [P], s[STARTED], a[id=Dq5I6QbYRRSURnPd5ND-tQ], failed_attempts[0]
    --------[.snapshot-blob-cache][0], node[tpz5rx_gRRa3OjkJI6mloA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=8u-WV5b5Rn6_I4Na5RtgCg], unassigned_info[[reason=INDEX_CREATED], at[2022-12-23T08:48:09.808Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]

    -- index [[irtdwbggqk/EWy0wlDNTp-SuphMLpChBg]]
    ----shard_id [irtdwbggqk][0]
    --------[irtdwbggqk][0], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=ZLZF9yuJR-29s2fVV6-cTA], failed_attempts[0], expected_shard_size[225]
    ----shard_id [irtdwbggqk][1]
    --------[irtdwbggqk][1], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=X_3UKgoKSmmag8rpPyToEg], failed_attempts[0], expected_shard_size[12554]
    ----shard_id [irtdwbggqk][2]
    --------[irtdwbggqk][2], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=qy3S2v6gQnaa6GOBMvyBBA], failed_attempts[0], expected_shard_size[4715]
    ----shard_id [irtdwbggqk][3]
    --------[irtdwbggqk][3], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=H2gYMJcNRXqzUVTMK5tkxQ], failed_attempts[0], expected_shard_size[4651]
    ----shard_id [irtdwbggqk][4]
    --------[irtdwbggqk][4], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=WCn9ZsfrT32NuWSI3-AzMA], failed_attempts[0], expected_shard_size[4835]
    ----shard_id [irtdwbggqk][5]
    --------[irtdwbggqk][5], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=JHagdX-gQiyX5MmramHavQ], failed_attempts[0], expected_shard_size[225]
    ----shard_id [irtdwbggqk][6]
    --------[irtdwbggqk][6], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=bCTRUZNuTmObXnO9eJ8f-Q], failed_attempts[0], expected_shard_size[225]
    ----shard_id [irtdwbggqk][7]
    --------[irtdwbggqk][7], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=zERtsl2LS56brRGFEMfgag], failed_attempts[0], expected_shard_size[225]
    ----shard_id [irtdwbggqk][8]
    --------[irtdwbggqk][8], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=041Mq4PQRUGbGe4EJPdecA], failed_attempts[0], expected_shard_size[10574]
    ----shard_id [irtdwbggqk][9]
    --------[irtdwbggqk][9], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=FD767KG-TV65CkiMMb8D-g], failed_attempts[0], expected_shard_size[4651]

    routing_nodes:
    -----node_id[eBGx5PwCQ42xJq8vqRKqyw][V]
    --------[.snapshot-blob-cache][0], node[eBGx5PwCQ42xJq8vqRKqyw], [P], s[STARTED], a[id=Dq5I6QbYRRSURnPd5ND-tQ], failed_attempts[0]
    -----node_id[tpz5rx_gRRa3OjkJI6mloA][V]
    --------[.snapshot-blob-cache][0], node[tpz5rx_gRRa3OjkJI6mloA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=8u-WV5b5Rn6_I4Na5RtgCg], unassigned_info[[reason=INDEX_CREATED], at[2022-12-23T08:48:09.808Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]
    -----node_id[BQxMyRTARK--skKRDnj97A][V]
    --------[irtdwbggqk][0], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=ZLZF9yuJR-29s2fVV6-cTA], failed_attempts[0], expected_shard_size[225]
    --------[irtdwbggqk][2], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=qy3S2v6gQnaa6GOBMvyBBA], failed_attempts[0], expected_shard_size[4715]
    --------[irtdwbggqk][4], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=WCn9ZsfrT32NuWSI3-AzMA], failed_attempts[0], expected_shard_size[4835]
    --------[irtdwbggqk][6], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=bCTRUZNuTmObXnO9eJ8f-Q], failed_attempts[0], expected_shard_size[225]
    --------[irtdwbggqk][8], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=041Mq4PQRUGbGe4EJPdecA], failed_attempts[0], expected_shard_size[10574]
    --------[irtdwbggqk][9], node[BQxMyRTARK--skKRDnj97A], [P], s[STARTED], a[id=FD767KG-TV65CkiMMb8D-g], failed_attempts[0], expected_shard_size[4651]
    -----node_id[RyaaHVHRQne1t5gJMiOrxQ][V]
    --------[irtdwbggqk][1], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=X_3UKgoKSmmag8rpPyToEg], failed_attempts[0], expected_shard_size[12554]
    --------[irtdwbggqk][3], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=H2gYMJcNRXqzUVTMK5tkxQ], failed_attempts[0], expected_shard_size[4651]
    --------[irtdwbggqk][5], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=JHagdX-gQiyX5MmramHavQ], failed_attempts[0], expected_shard_size[225]
    --------[irtdwbggqk][7], node[RyaaHVHRQne1t5gJMiOrxQ], [P], s[STARTED], a[id=zERtsl2LS56brRGFEMfgag], failed_attempts[0], expected_shard_size[225]
    ---- unassigned
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown(PartiallyCachedShardAllocationIntegTests.java:325)

    java.lang.AssertionError: test leaves persistent cluster metadata behind
    Expected: an empty collection
         but: <[cluster.routing.rebalance.enable]>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:570)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2242)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:833)

@albertzaharovits
Copy link
Contributor

Another failure: https://gradle-enterprise.elastic.co/s/5l6rzi4zcg4xu

DaveCTurner added a commit that referenced this issue Jan 4, 2023
@DaveCTurner
AwaitsFix for #91800
f07824b
@hendrikmuhs
Copy link

another one: https://gradle-enterprise.elastic.co/s/ef5krfb6tfbo4

@cbuescher
Copy link
Member

another today: https://gradle-enterprise.elastic.co/s/nk3zokpfsdkz4

@csoulios
Copy link
Contributor

csoulios commented Jan 26, 2023 •
edited
Loading

Failed again today, this time in 8.6 branch
https://gradle-enterprise.elastic.co/s/a2tssxkqs4rdw

org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests > testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown FAILED
    java.lang.AssertionError: balanced across [node_s1, node_s2] in cluster uuid: XOeu58CURheXJ9gAiyfGVw [committed: false]
    version: 165
    state uuid: NSmB7Bo7Q0yXnUrovgbn5g
    from_diff: false
    meta data version: 0
       coordination_metadata:
          term: 1
          last_committed_config: VotingConfiguration{bFDeT28pQd68V5ODTHLaig}
          last_accepted_config: VotingConfiguration{bFDeT28pQd68V5ODTHLaig}
          voting tombstones: []
    metadata customs:
       index-graveyard: IndexGraveyard[[]]
    nodes: 
       {node_s2}{z7h6fcEITzKyRSjeMYkkmw}{xzcStB_BQQaP3d6NsJo9sQ}{node_s2}{127.0.0.1}{127.0.0.1:18882}{d}{xpack.installed=true}
       {node_s1}{x7hMqw8jQFGfmoIOKFTVvg}{fJPyCqkgTFCzBzOk_10Y2A}{node_s1}{127.0.0.1}{127.0.0.1:18883}{d}{xpack.installed=true}
       {node_s0}{bFDeT28pQd68V5ODTHLaig}{Z8gr8mD0ScStuVjD014v8g}{node_s0}{127.0.0.1}{127.0.0.1:18881}{cdfhilmrstw}{xpack.installed=true}, local, master
    routing_table (version 88):
    -- index [[agfcennvsf/0tF76Bo3RrO7Y-EjlEaixg]]
    ----shard_id [agfcennvsf][0]
    --------[agfcennvsf][0], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=63I2iceDTMqtzWR6YVn7Uw], failed_attempts[0], expected_shard_size[12029]
    ----shard_id [agfcennvsf][1]
    --------[agfcennvsf][1], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=s2QHEnAERnexRUJtH3rraw], failed_attempts[0], expected_shard_size[5177]
    ----shard_id [agfcennvsf][2]
    --------[agfcennvsf][2], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=HT59xkd-Sb-T7kDSiclF-Q], failed_attempts[0], expected_shard_size[8217]
    ----shard_id [agfcennvsf][3]
    --------[agfcennvsf][3], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=B8r58b_FSJC5BAshnmSmCw], failed_attempts[0], expected_shard_size[5305]
    ----shard_id [agfcennvsf][4]
    --------[agfcennvsf][4], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=GeWIljnxQWWSjwLFUsm7hg], failed_attempts[0], expected_shard_size[6940]
    ----shard_id [agfcennvsf][5]
    --------[agfcennvsf][5], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=FQRoXSZJT0SbZ9cPObctPQ], failed_attempts[0], expected_shard_size[7092]
    ----shard_id [agfcennvsf][6]
    --------[agfcennvsf][6], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=t4oy6xk5RWybrajr2PP29A], failed_attempts[0], expected_shard_size[7092]
    ----shard_id [agfcennvsf][7]
    --------[agfcennvsf][7], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=M4NTWTksSumP6HgS1ddXpw], failed_attempts[0], expected_shard_size[5121]
    ----shard_id [agfcennvsf][8]
    --------[agfcennvsf][8], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=1qIeN2owRu2azCJuJxIb4A], failed_attempts[0], expected_shard_size[8217]
    ----shard_id [agfcennvsf][9]
    --------[agfcennvsf][9], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=EoIzZYbxRnyIC-5HFrhCgw], failed_attempts[0], expected_shard_size[11570]

    -- index [[.snapshot-blob-cache/wa5Lc-iJQ86PV7cBtWNjlQ]]
    ----shard_id [.snapshot-blob-cache][0]
    --------[.snapshot-blob-cache][0], node[z7h6fcEITzKyRSjeMYkkmw], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=AK0w6SXrRGGN_Tkgg4TlnA], unassigned_info[[reason=INDEX_CREATED], at[2023-01-26T08:16:47.858Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]
    --------[.snapshot-blob-cache][0], node[bFDeT28pQd68V5ODTHLaig], [P], s[STARTED], a[id=Gd2zssnRT8G5F7nQcH1ChA], failed_attempts[0]

    routing_nodes:
    -----node_id[z7h6fcEITzKyRSjeMYkkmw][V]
    --------[agfcennvsf][0], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=63I2iceDTMqtzWR6YVn7Uw], failed_attempts[0], expected_shard_size[12029]
    --------[agfcennvsf][2], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=HT59xkd-Sb-T7kDSiclF-Q], failed_attempts[0], expected_shard_size[8217]
    --------[agfcennvsf][4], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=GeWIljnxQWWSjwLFUsm7hg], failed_attempts[0], expected_shard_size[6940]
    --------[agfcennvsf][6], node[z7h6fcEITzKyRSjeMYkkmw], [P], s[STARTED], a[id=t4oy6xk5RWybrajr2PP29A], failed_attempts[0], expected_shard_size[7092]
    --------[.snapshot-blob-cache][0], node[z7h6fcEITzKyRSjeMYkkmw], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=AK0w6SXrRGGN_Tkgg4TlnA], unassigned_info[[reason=INDEX_CREATED], at[2023-01-26T08:16:47.858Z], delayed=false, allocation_status[no_attempt]], failed_attempts[0]
    -----node_id[x7hMqw8jQFGfmoIOKFTVvg][V]
    --------[agfcennvsf][1], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=s2QHEnAERnexRUJtH3rraw], failed_attempts[0], expected_shard_size[5177]
    --------[agfcennvsf][3], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=B8r58b_FSJC5BAshnmSmCw], failed_attempts[0], expected_shard_size[5305]
    --------[agfcennvsf][5], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=FQRoXSZJT0SbZ9cPObctPQ], failed_attempts[0], expected_shard_size[7092]
    --------[agfcennvsf][7], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=M4NTWTksSumP6HgS1ddXpw], failed_attempts[0], expected_shard_size[5121]
    --------[agfcennvsf][8], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=1qIeN2owRu2azCJuJxIb4A], failed_attempts[0], expected_shard_size[8217]
    --------[agfcennvsf][9], node[x7hMqw8jQFGfmoIOKFTVvg], [P], s[STARTED], a[id=EoIzZYbxRnyIC-5HFrhCgw], failed_attempts[0], expected_shard_size[11570]
    -----node_id[bFDeT28pQd68V5ODTHLaig][V]
    --------[.snapshot-blob-cache][0], node[bFDeT28pQd68V5ODTHLaig], [P], s[STARTED], a[id=Gd2zssnRT8G5F7nQcH1ChA], failed_attempts[0]
    ---- unassigned
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown(PartiallyCachedShardAllocationIntegTests.java:325)

    java.lang.AssertionError: test leaves persistent cluster metadata behind
    Expected: an empty collection
         but: <[cluster.routing.rebalance.enable]>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:569)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2237)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:833)

REPRODUCE WITH:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown" -Dtests.seed=A74C56EEA512A7D6 -Dtests.locale=de-CH -Dtests.timezone=America/Atikokan -Druntime.java=17

csoulios pushed a commit that referenced this issue Jan 26, 2023
@DaveCTurner @csoulios
AwaitsFix for #91800
db730c7
@idegtiarenko idegtiarenko mentioned this issue Aug 7, 2023
Merged
@idegtiarenko idegtiarenko self-assigned this Aug 7, 2023
@idegtiarenko
Copy link
Contributor

Should be resolved by the above change. I will reopen in case of new failures

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@idegtiarenko idegtiarenko

@DaveCTurner DaveCTurner

Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests