Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] PartiallyCachedShardAllocationIntegTests testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown failing #92206

Closed
iverase opened this issue Dec 7, 2022 · 2 comments
Closed

[CI] PartiallyCachedShardAllocationIntegTests testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown failing #92206

iverase opened this issue Dec 7, 2022 · 2 comments
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@iverase
Copy link
Contributor

iverase commented Dec 7, 2022

Build scan:
https://gradle-enterprise.elastic.co/s/5uxbnkiyboiwc/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests/testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown

Reproduction line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown" -Dtests.seed=DD68C82929ABEBE4 -Dtests.locale=ca-ES -Dtests.timezone=America/Santarem -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests&tests.test=testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown

Failure excerpt:

java.lang.AssertionError: balanced across [node_s2, node_s3] in cluster uuid: ZeeTFS5GQPicgmlLNplemg [committed: false]
version: 135
state uuid: sXCMsvr8So2kQnr_HuoKwQ
from_diff: false
meta data version: 0
   coordination_metadata:
      term: 1
      last_committed_config: VotingConfiguration{g_Rm5MD2Rqa1JRIA1VIApg}
      last_accepted_config: VotingConfiguration{g_Rm5MD2Rqa1JRIA1VIApg}
      voting tombstones: []
metadata customs:
   index-graveyard: IndexGraveyard[[]]
nodes: 
   {node_s2}{ODNt6JpXRnGQuJimnqpjBg}{WAd7SZD6SaOgo8HW2nMffg}{node_s2}{127.0.0.1}{127.0.0.1:27193}{d}{xpack.installed=true}
   {node_s3}{i_MpuK89RxygDxWUY7apmg}{7P9KWJwhQ2OaKJxuOTkUUQ}{node_s3}{127.0.0.1}{127.0.0.1:27194}{d}{xpack.installed=true}
   {node_s1}{g_Rm5MD2Rqa1JRIA1VIApg}{kk_JWqY4RFaCDDWRkmodeA}{node_s1}{127.0.0.1}{127.0.0.1:27192}{cdfhilmrstw}{xpack.installed=true}, master
   {node_s0}{Q6bq3wIaTES9_fl3NikG2A}{N3khPAMmRGeueMD_NI2xTw}{node_s0}{127.0.0.1}{127.0.0.1:27191}{cdfhilmrstw}{xpack.installed=true}
routing_table (version 75):
-- index [[.snapshot-blob-cache/3FpBdzszQi2Kxq8PSP26sg]]
----shard_id [.snapshot-blob-cache][0]
--------[.snapshot-blob-cache][0], node[Q6bq3wIaTES9_fl3NikG2A], [R], s[STARTED], a[id=m984jjkfQ86aMu1LJgWZyw], failed_attempts[0]
--------[.snapshot-blob-cache][0], node[g_Rm5MD2Rqa1JRIA1VIApg], [P], s[STARTED], a[id=vEFL0uAqSC-Vx4gywkajtg], failed_attempts[0]

-- index [[xewqwyfoxg/s5hY1V3uTGiVEu7adt7gPA]]
----shard_id [xewqwyfoxg][0]
--------[xewqwyfoxg][0], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=KF34NKLvS9Sp75qGXDwzrQ], failed_attempts[0], expected_shard_size[5254]
----shard_id [xewqwyfoxg][1]
--------[xewqwyfoxg][1], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=MEH372SFT8i7di02SDKxdA], failed_attempts[0], expected_shard_size[6519]
----shard_id [xewqwyfoxg][2]
--------[xewqwyfoxg][2], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=R48xejJHRliPrq7h0lVo0g], failed_attempts[0], expected_shard_size[16117]
----shard_id [xewqwyfoxg][3]
--------[xewqwyfoxg][3], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=Eiv8hiVkTaGgg2bFvSu9WA], failed_attempts[0], expected_shard_size[4194]
----shard_id [xewqwyfoxg][4]
--------[xewqwyfoxg][4], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=vn_0zkM5TxyQf61X-9onKg], failed_attempts[0], expected_shard_size[4042]
----shard_id [xewqwyfoxg][5]
--------[xewqwyfoxg][5], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=iFJc4tB0RIyXUi0aSlA4bA], failed_attempts[0], expected_shard_size[8183]
----shard_id [xewqwyfoxg][6]
--------[xewqwyfoxg][6], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=9wAz6GTIRpOMfs8DkEtJDw], failed_attempts[0], expected_shard_size[6559]
----shard_id [xewqwyfoxg][7]
--------[xewqwyfoxg][7], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=eBpFTWs_RHO_RoH0Kqe-Fw], failed_attempts[0], expected_shard_size[5254]
----shard_id [xewqwyfoxg][8]
--------[xewqwyfoxg][8], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=4vtStpp4TteMHB6ythaLYw], failed_attempts[0], expected_shard_size[4042]
----shard_id [xewqwyfoxg][9]
--------[xewqwyfoxg][9], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=DkwjqZRrSV6bYxd9PRkPlA], failed_attempts[0], expected_shard_size[225]

routing_nodes:
-----node_id[Q6bq3wIaTES9_fl3NikG2A][V]
--------[.snapshot-blob-cache][0], node[Q6bq3wIaTES9_fl3NikG2A], [R], s[STARTED], a[id=m984jjkfQ86aMu1LJgWZyw], failed_attempts[0]
-----node_id[ODNt6JpXRnGQuJimnqpjBg][V]
--------[xewqwyfoxg][0], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=KF34NKLvS9Sp75qGXDwzrQ], failed_attempts[0], expected_shard_size[5254]
--------[xewqwyfoxg][2], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=R48xejJHRliPrq7h0lVo0g], failed_attempts[0], expected_shard_size[16117]
--------[xewqwyfoxg][4], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=vn_0zkM5TxyQf61X-9onKg], failed_attempts[0], expected_shard_size[4042]
--------[xewqwyfoxg][6], node[ODNt6JpXRnGQuJimnqpjBg], [P], s[STARTED], a[id=9wAz6GTIRpOMfs8DkEtJDw], failed_attempts[0], expected_shard_size[6559]
-----node_id[i_MpuK89RxygDxWUY7apmg][V]
--------[xewqwyfoxg][1], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=MEH372SFT8i7di02SDKxdA], failed_attempts[0], expected_shard_size[6519]
--------[xewqwyfoxg][3], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=Eiv8hiVkTaGgg2bFvSu9WA], failed_attempts[0], expected_shard_size[4194]
--------[xewqwyfoxg][5], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=iFJc4tB0RIyXUi0aSlA4bA], failed_attempts[0], expected_shard_size[8183]
--------[xewqwyfoxg][7], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=eBpFTWs_RHO_RoH0Kqe-Fw], failed_attempts[0], expected_shard_size[5254]
--------[xewqwyfoxg][8], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=4vtStpp4TteMHB6ythaLYw], failed_attempts[0], expected_shard_size[4042]
--------[xewqwyfoxg][9], node[i_MpuK89RxygDxWUY7apmg], [P], s[STARTED], a[id=DkwjqZRrSV6bYxd9PRkPlA], failed_attempts[0], expected_shard_size[225]
-----node_id[g_Rm5MD2Rqa1JRIA1VIApg][V]
--------[.snapshot-blob-cache][0], node[g_Rm5MD2Rqa1JRIA1VIApg], [P], s[STARTED], a[id=vEFL0uAqSC-Vx4gywkajtg], failed_attempts[0]
---- unassigned

  at org.junit.Assert.fail(Assert.java:88)
  at org.junit.Assert.assertTrue(Assert.java:41)
  at org.elasticsearch.xpack.searchablesnapshots.cache.shared.PartiallyCachedShardAllocationIntegTests.testPartialSearchableSnapshotDelaysAllocationUntilNodeCacheStatesKnown(PartiallyCachedShardAllocationIntegTests.java:325)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@iverase iverase added :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >test-failure Triaged test failures from CI labels Dec 7, 2022
@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Dec 7, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@iverase iverase mentioned this issue Dec 7, 2022
Merged
@pxsalehi
Copy link
Member

pxsalehi commented Dec 7, 2022

Duplicate of #91800

@pxsalehi pxsalehi marked this as a duplicate of #91800 Dec 7, 2022
@pxsalehi pxsalehi closed this as completed Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pxsalehi @iverase @elasticsearchmachine